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ABSTRACT 

A  method  for  estimating  the  reliability  of  maintenance  performance  is  developed 
and  applied  to  tasks  involved  in  scheduled  maintenance  for  Titan  II  engines.  The 
approach  involves  the  combined  use  of  ratings  and  empirically  derived  reliability 
figures.  A  modification  of  the  design  engineer's  redundancy  formula  is  developed 
for  estimating  the  increase  in  human  reliability  achieved  when  two  mechanics  work 
together  in  the  pcrfoi-mance  of  a  single  maintenance  task. 

This  study  demonstrates  that  highly  consistent  ratings  of  task-element - 
reliability  can  be  obtained  from  groups  of  qualified  raters. 

Plans  for  validating  the  human  reliability  estimates  obtained  during  Category  II 
testing  at  Vandonberg  Air  Force  Base  are  described.  Suggestions  for  further  research 
and  application  of  the  findings  are  given. 


ii 


Esport  LRP  317/TDR- 63-218 


TABLE  OF  CONTENTS 


Z,  Introduction 

XI.  Summary 

A .  Method 

B.  Results 

C.  Suggested  Applications 

III.  Development  of  the  Method 

A.  Previous  Efforts 

B.  Method  of  Obtaining  Ratings 

C.  Empirically  Based  Performance  Reliability  Estimates 

IV.  The  Relationship  Between  the  Ratings  and  the  Erapirically  Based 
Reliability  Estimates 

V.  Taking  Personnel  Redundancy  into  Account  in  Estimating  Task 

Reliabilities 

VI.  Converting  Task-Element  ReliabilitifS  Into  Task  Reliabilities 

VII.  A  Comparison  of  Reliability  Estiraat.es  with  Air  Force  Maintenance 
Records 

VIII.  Plan  for  Validation  of  Reliability  Estimates 

IX.  Suggestions  for  Further  Research 

A.  Determining  Preventive  Maintenance  Schedules 

B.  More  Extensive  Coverage  of  Maintenance  Tasks 

C.  More  Elaborate  Treatment  of  Dependency 

D.  More  Extensive  Treatment  of  Consequences  of  Errors 

E.  Development  of  Methods  for  Rapidly  Estimating  Task 
Reliability 

References 

Glossary 


APPENDIX 

Instructions  for  Raters  Used  in  Preliminary  Study 


Page 

1 

1 

2 

L, 

10 

12 

12 

17 

39 

47 

50 

5k 

55 

56 
56 

58 

59 
59 

61 

62 
6k 


iii 


Report  LRP  317/TDR- 63-218 


TABLE  LIST 

Table 

Ss® 

1 

Estimated  Reliability  of  the  Personnel  Subsystem  in  the 
Performance  of  Titan  II  Maintenance  Tasks 

6 

2 

Task  Elements  in  Task  "Prepare  for  Leak  Check  of  Turbo¬ 
pump-Fuel  Pump  Seal" 

19 

3 

Measurement  Reliability  of  Ratings,  Preliminary  Study 

23 

4 

Task  Element  Reliabilities  from  Data  Store 

Zk 

5 

Measurement  Reliability  of  Ratings  —  33  Raters 

29 

6 

Measurement  Reliability  of  Ratings  —  18  A(3C  Engineers 

30 

7 

Measurement  Reliability  of  Ratings  —  6  AGO  Technicians 

31 

8 

Measurement  Reliability  of  Ratings  —  9  Vandenberg  Air 

Force  Base  Missile  Engine  Mechanics 

32 

9 

Task  Element  Reliability  Estimates 

35 

10 

Intsrcorrelations  Among  Mean  Ratings  of  Task  Elements  by 
Different  Groups  of  Raters 

4o 

11 

Derivations  of  Task  Reliability  from  Task  Element  Relia¬ 
bilities 

53 

FICJRE  LIST 

Figure 

Page 

1 

Sorting  Arrangement 

22 

2 

Scatter  Plot  of  Ratings  and  Data  Store  Error  Rates, 
Preliminary  Study 

26 

3 

i^Jean  Freque:icy  with  which  Ratings  Were  Assigned  to 

Categories 

38 

•? 

Scatter  Plot  of  Ratings  and  Data  Store  Error  Rates 

1j8 

5 

boatte."  irlot  of  Ratings  and  Data  Store  Error  Rates, 

Serai -Icgaiithraic  Paper 

k9 

6 

Individual  Summary  Form 

57 

iv 


Report  LRP  317/TDR- 63-218 


I.  INTRODUCTION 

The  objective  of  this  study  vas  to  develop  a  means  of  predicting  personnel 
effectiveness  during  scheduled  checkout  and  maintenance  activities  performed  on  the 
Titan  II  Propulsion  System.  It  was  recognized  that  the  inherent  capability  designed 
into  the  Titan  II  Propulsion  System  could  not  be  realized  f\illy  unless  due  considera¬ 
tion  was  given  to  the  performance  of  the  personnel  who  service  and  maintain  the  sys¬ 
tem. 


In  the  development  of  aerospace  systems,  the  United  States  Air  Force  considers 
the  human  component  as  the  personnel  subsystem.  The  personnel- subsys’oem  concept  en¬ 
ables  the  Air  Force  to  pursue  a  develojaaent  and  test  program  for  the  human  component 
similar  to  that  for  the  hardware  components.  In  the  past,  reliability  predictions 
for  personnel  performance  were  not  employed  in  the  development  of  weapon  systeras, 
and  decisions  regarding  these  activities  were  based  on  generalized  ratiiigs  such  as 
"good,"  "average,"  "fair,"  or  "poor,"  In  their  efforts  to  provide  a  highly  effective 
man/machine  system,  design  engineers,  technical  writers,  and  training  personnel  were 
hampered  by  the  vague  nature  of  such  performsmce  descriptions. 

II,.  SUMMPJ^Y 

Quantitative  reliability  information  was  available  for  hardware  components  so 
that  decisions  and  system  integration  could  be  made  in  a  technically  sound,  wsay. 
Designers,  for  example,  were  able  to  choose  gages,  valves,  controls,  and  similar 
components  by  examining  their  reliability  as  well  as  other  pertinent  characteristics. 
Efforts  were  made  to  develop  techniques  for  predicting  and  eveJ.uating  the  reliability 
of  the  humein  component  in  a  manner  similar  to  that  which  proved  so  useful  in  hardware 
research  and  development. 

The  availability  of  quaiititative  human  reliability  information  should  aid  de¬ 
cision  making  in  each  of  the  following  areas; 

1.  Designing  the  propulsion  system  and  supporting  equipment  for  ease 
of  operation  and  maintenance. 
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II,  Summary  (cont.) 

2.  Providing  clear  and  effective  Checklists  and  Technical  Orders. 

3.  Providing  effective  inputs  to  the  training  of  Titan  II' Missile 
Eng;ine  Mechsuaics. 

While  actions  in  these  three  areas  will  make  a  substantial  contribution  toward 
achieving  the  highest  possible  human  reliability  under  operationeil  conditions;  this 
reliability  is  clearly  a  function  of  additional  factors,  including  such  critical 
cleteiminants  as  attitudes,  feelings,  and  motivations  of  Air  Force  mechanics. 

This  report  describes  the  develojanent  of  a  method  for  estimating  the  relia¬ 
bility  of  maintenance  perfoiroance.  The  reader  should  bear  in  mind  that  the  method 
has  not  been  validated  using  Titan  II  operationed  performance  data.  Until  the  re¬ 
sults  of  present  validation  efforts  stre  available,  the  performance  reliability  esti¬ 
mates  should  be  cautiously  applied  as  they  were  derived  from  subjective  Judgements 
and  limited  empirical  data.  Because  the  empirical  data  were  obtained  relatively 
early  in  the  research  and  development  cycle,  it  is  highly  probable  that  the  opera¬ 
tional-reliability  figures  will  be  higher  them  those  reported.  Even  at  that  eeurly 
data,  the  task  failure  rate  of  l4,4^  was  close  to  the  10  to  15^  figure  t^/picaJL  of 
the  defense  electronics  industry  as  estimated  by  the  Americeui  Institute  for  Research 
(Ref  1) . 

A.  METHOD 

A  promising  method  for  estimating  the  reliability  of  maintenance  perfor¬ 
mance  has  been  developed  and  applied  to  Titan  II  Propulsion  System  scheduled  main¬ 
tenance  activities.  The  method  involves  the  following  steps; 

1.  Specify  the  maintenance  tasks  to  be  performed,  e.g.,  service  oil 
sump  of  turbopump  gear  box,  perform  functional  check  of  thrust-chamber  valves,  per¬ 
form  leak  check  of  fuel  system,  etc. 


.M  ^  * 
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:4i/itkm3aEBM0JUMSi^MfJt^ll^fa 


Report  LRP  317/TDR-63-213 


II,  A,  Method  (cont.) 


2.  Identify  the  task  elements  that  must  he  performed  to  accon5>lish  the 
total  tasks,  e.g.,  verify  s'ffitch  position,  connect  flexible  hose,  read  time  (Brush 
Recorder),  install  lockwire,  etc. 

3.  Prom  judges  familiar  with  maintenance  tasks  and  typical  Air  Force 
mechanics,  obtain  rating  for  likelihood  of  error  in  perfonning  the  task  elements. 

In  the  present  study,  the  33  rating  judges  were  iS  Aerojet-General  Personnel  Subsys¬ 
tem  and  Reliability  Engineers,  6  Aerojet-General  technicians,  and  9  Vandenberg  APB 
missile-engine  mechanics.  More  consistent  ratings  were  obtained  from  the  engineers 
than  from  either  the  technicians  or  the  mechanics.  The  degree  of  inter-rate  agree¬ 
ment  among  the  l8  engineers  was  very  high.  Only  ratings  from  the  engineers  were 
used  in  the  subsequent  analyses. 

U.  Obtain  empirically  based  reliability  estimates  for  at  least  some  of 
the  task  elements .  In  the  present  study,  29  empirically  based  reliability  estimates 
were  derived  from  the  Payne-Altman  Index  of  Electronic  Equipment  Operability  (Ref  l). 
The  Payne-Altman  figures  are  extrapolations  to  field  conditions  based  on  the  results 
of  laboratory  studies  available  in  the  experimental  literature.  Using  results  of 
the  Titan  II  Personnel  Subsystem  Test  and  Evaluation  Studies,  conducted  in  a  simulated 
silo  in  Sacramento,  a  new  conversion  factor  as  developed  and  applied  to  the  index 
data  to  make  experimental  findings  more  compatible  with  Titan  II  operating  conditions. 


5.  Prepare  a  scatter  diagram  for  the  task  elements  for  which  both  rat¬ 
ings  and  empirically  based  reliability  estimates  are  available.  I  it  a  regression 
equs,tion  to  the  data  and  derive  reliability  estimates  for  the  task  el'  ments  for  which 
only  ratii  gs  are  available  (Table  S).  In  the  present  study,  the  correlation  of  the 
task-element  ratings  and  modified  reliability  estimates  of  the  Index  of  Electronic 
Equipment  Operability  was  indicating  a  significant  relationship.  In  deriving 

reliability  estimates,  a  logarithmic  curve  was  applied  to  the  data  because  it  pro¬ 
vided  a  better  fit  rather  than  a  linear  regression  line  and  was  consistent  wi,th 
previous  findings  in  the  study  of  rating  phenomena. 
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ir,  A,  Method  (cont.) 

6.  If  a  second  mechanic  vlll  he  available  to  assist  the  first  one  in 
the  performance  of  the  task,  adjust  the  task  element  reliability  estimates  to  take 
account  of  this  redundancy. 

7.  V  Working  from  a  detailed  set  of  procedural  instructions,  determine 
the  task  elements  involved  in  performing  each  maintenance  task.  Record  the  appropri¬ 
ate  reliability  estimate  for  each  task  element. 

8.  -Determine  the  Task-performance  reliability  by  canputing  the  product 
of  the  separate  task-element  reliabilities. 

The  method  has  logicsd  validity,  and  its  results  are  supported  by  the 
limited  empirical  evidence  presently  available.  Task-element  reliabilities  derived 
by  this  method  are  similar  to  some  empirically  based  reliability  figures  acquired 
from  records  of  aircraft  meintenance.  While  such  evidence  is  recognized  as  limited 
in  quantity,  its  confirmatcry  nature  strengthens  confidence  in  the  method  described 
and  justifies  a  further  investigation  and  a  possible  increase  in  range  of  application. 
The  final  test  of  the  methoi  will  involve  a  comparison  of  the  present  predictions 
with  the  human  reliability  measures  obtained  during  Category  II  testing  at  Vanderiberg 
Air  Force  Base.  Plans  for  such  a  validation  study  are  described.  The  report  is  con¬ 
cluded  with  suggestions  for  further  research. 

B.  RESULTS 

The  results  of  the  study  are  presented  in  Table  1  and  consists  of  the 
human  reliability  estimates  for  the  64  scheduled  maintenance  tasks  of  the  Titan  II 
engine.  The  lover  figure  in  Table  1  is  the  reliability  estimate  for  the  task  when 
individually  performed.  The  higher  figure  is  the  reliability  for  the  task  when  two 
mechanics  are  present  in  the  rilo  and  redundancy  is  considered.  Tasks  having  both 
relatively  low  predicted  human  reliability  and  high  criticality  should  receive 
priority  attention.  Tasks^  that  are  eimong  the  lowest  third  in  redundant  reliability 
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II,  B,  Results  (cont.) 

of  all  schedule  maintenance  tasks  and  having  a  criticality  rating  of  3  are  starred 
(*)  in  Table  1.  To  aid  in  interpreting  the  significance  of  the  reliability  estimates, 
criticeJ-ity  ratings  from  Type  II  Basic  Kata  are  included  in  Table  1.  The  criticality 
code  is  defined  below. 

It  should  be  noted  that  the  figures  in  Table  1  are  estimates  of  the  re¬ 
liability  of  the  personnel  subsystem  and  are  not  estimates  of  the  reliability  of  the 
propulsion  subsystem  developed  by  Aerojet-General.  The  reliability  of  the  propul¬ 
sion  subsystem  is  considerably  higher  than  the  estimated  reliability  of  the  personnel 
subsystem. 


In  addition,  the  figures  should  not  be  interpreted  to  mean  that  any  in-- 
dication  of  the  lack  of  personnel  reliability  entered  in  Table  1  will  result  in 
weapon- system  failiire  cr  extreme  safety  hazard.  Many  of  the  performance  errors  will 
be  less  critical,  res’ulting  only  in  increased  maintenance  time  or  excessive  consump¬ 
tion  of  spare  parts. 

Criticality  Code  Definition 

2  Tasks  critical  for  subsystem  operation  that  may 

result  in  some  system  degradation  if  not  correctly 
performed.  Such  tasks  fall  into  the  category  of 
things  that  affect  equipment  that  is  nice  to  have 
or  that  the  system  would  be  more  effective  if  the 
task  had  been  performed  correctly  but  the  mission 
can  succeed  by  using  alternate  modes,  manual  con- 
control. 

3  Tasks  that  are  critical  to  system  operation  must 

be  performed  correctly.  If  not  performed  correctly, 
they  may  prevent  the  system  from  working  or  reduce 
operational  effectiveness  to  an  unacceptable  level. 
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TABLE  1 

ESTIMATED  EELIABILITy  OF  THE  PERSONHEL  SUBSYSTEM 
PERFORMANCE  OF  TITAIJ  II  MAINTENANCE  TASKS 


Hmnan  Reliability 
Indi-  Redun- 


Performance  Task 

Stage 

vidxial 

dant 

Prepare  for  Lesdc  Check  of  Thrust  Chamber 
and  Lines  Below  Thrust- Chamber  Valves 

II 

.6712 

.7322 

Prepare  for  Leak  Check  of  Thrust  Chamber 
and  Lines  Below  Thrust- Chamber  Valves 

I 

.6736 

.7^*22 

^Prepare  for  Functional  Check  of  Thrust- 
Chamber  Valves 

II 

.6856 

.7581 

*Perform  Functional  Check  of  (Thrust-Chamber 
Valves 

I 

.6933 

.7658 

Prepare  for  Leak  Check  of  Subassembly  Hot- 
Gas  System  and  Turbine  Seal 

I 

.7014 

.7657 

*Leak-Check  Installation  of  Pressurization 

Kit 

II 

.7526 

.8228 

*Leak- Check  Installation  of  Pressurization 

Kit 

I 

.7531 

.8219 

^Perform  Leak  Check  of  Hot-Gas  System  and 
Turbine  Seal 

II 

.7644 

.8523 

*Perform  Leak  Check  of  Turbopump  Oxidizer 
Gearbox  Seal 

II 

.7678 

.8240 

^♦'Perforra  Leak  Check  of  Turbopump  Fuel  Pump 
Seal 

II 

.7680 

.8194 

^Perform  Leak  Check  of  Turbopump  Fuel  Gear- 
•  box  Seal 

II 

.7761 

.8242 

^Perform  Electrical  Check 

I 

.7838 

.8361 

^Perform  Leak  Check  of  Turbopump  Oxidizer 

Pump  Seal 

II 

.7906 

.8403 

^Perform  Leak  Check  of  Thrust  Chamber  end 
Lines  Below  Thrust- Chamber  Valves 

II 

.8228 

.8699 

*Perform  Leak/Functional  Check  of  Thrust- 
Chamber  Pressure  Switch 

I 

.8234 

.8809 

^Perform  Leak  Check  of  Subassembly  Hot-Gas 
System  and  Turbine  Seed 

I 

,8271 

.8734 

Criti- 

ced-lty. 

2 
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IIABLE  1  (cont.) 


Human  Reliatilit 


Stage 

Indi¬ 

vidual 

Redun¬ 

dant 

Prepare  for  Leak  Check  of  Hot-Gas  System 
and  Inrbine  Seal 

II 

.8P.76 

,8696 

^Prepare  for  Leak  Check  of  Suhassemhly 
Turbopump  F>ael  Pump  Seal 

I 

.6337 

.8852 

*Perform  Leak  Check  of  Thrust  Chamber  and 
Lines  Belo^r  Thrust- Chamber  Valves 

I 

.3402 

.8820 

^Perform  Leak  Check  of  Subassembly  Turbo- 
purap  Fuel  Pump  Se^ 

I 

.8450 

.8762 

"Prepare  for  Leak  Check  of  Subassembly 
Turbopump  Oxidizer  Pump  Seal 

X 

X 

.8452 

.8920 

Prepare  for  Leak  Check  of  Subassembly  Fuel 
System 

I 

.8457 

.8928 

^Perform  Leak  Check  of  Subassembly  Fuel 
System 

I 

.8465 

.8814 

Perform  Leak  Check  of  Subassembly  Turbo- 
pvimp  Fuel  Gearbox  Seal 

I 

.8513 

.8941 

Perform  Leedj  Check  of  Subassembly  Turbo¬ 
pump  Oxidizer  Gearbox  Seal 

I 

.8513 

.8941 

Perform  Electrical  Check 

II 

.8553 

.8918 

Perform  Leak  Check  of  Subassembly  Turbo¬ 
pump  Oxidizer  Pump  Seal 

II 

:8650 

:9146 

Prepare  for  Leak  Check  of  Subassembly 
Turbopump  Oxidizer  Gearbox  Seal 

I 

.8750; 

.9163 

Prepare  for  Leak  Check  of  Turbopump  Fuel 
Pump  Seal 

II 

.8740 

.9163 

Prepare  for  Leak  Check  of  Subassembly 
Turbopump  Fuel  Gearbox  Seal 

I 

.8751 

.9098 

Prepare  for  Leak  Check  of  Subassembly 
Oxidizer  System 

I 

.8775 

.9211 

Prepare  for  Leeik  Check  of  Turbopump  Fuel 
Gearbox  Seal 

II 

.8785 

.9137 

Prepare  for  Leak  Check  of  Turbopump  Oxidi- 

II 

.8801 

.9134 

zer  Pump  Seed 
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Hianan  Reliability 


Stage 

Indi¬ 

vidual 

Redun¬ 

dant 

Criti¬ 

cality 

Prepare  for  Thrust- Chamber  Valves  Sanc¬ 
tioned.  Check 

I 

.8811 

.9069 

2 

Prepare  for  Thrust- Chamber  Valves  Func¬ 
tional  Check 

I 

.881H 

.9085 

3 

Perform  Leak  Check  of  Subassembly  Oxidizer 
System 

I 

.8899 

.9206 

3 

Perform  Leak  Check  of  Oxidizer  Autogenous 
System  Between  Interface  and  Superheat 

Burst  Diaphragm 

I 

.8914 

.9207 

3 

Prepare  for  Leak  Check  of  Turbopimp  Oxidi- 
dizer  Gearbox  Seal 

II 

.8916 

.9253 

2 

Perform  Leedc  Check  of  Fuel  System 

II 

.8962 

.9264 

3 

Perform  Engine  Electrical- Conduit  Leak 

Check 

II 

.8993 

.9303 

3 

Prepeire  for  Electrical- Conduit  Leak  Check 

II 

.9032 

.9312 

2 

Prepare  for  FJ.ectrical- Conduit  Leak  Check 

I 

.9130 

.9388 

2 

Prepare  for  Leak  Check  of  Oxidizer  Autog¬ 
enous  System  Between  Interface  and  Super¬ 
heater  Burst  Diaphragm 

I 

.9051 

.9357 

2 

Perform  Engine  Electrlceil- Conduit  Leak 

Check 

I 

.9083 

.9318 

3 

Prepare  for  Thrust-Chamber  Switch  Leak/ 
Functional  Check 

I 

.9084 

.9350 

3 

Prepare  for  Leedc  Check  of  Fuel  System 

II 

.9085 

.9361 

2 

Prepare  for  Leeik  Check  of  Oxidizer  System 

II 

.9095 

.9376 

2 

Install  Gearbox  Pressurization  Kit 

II 

.9198 

.9455 

3 

Perform  Leak  Check  of  Oxidizer  System 

II 

.9200 

.9436 

3 

Prepare  for  Electrical  Check 

I 

.9210 

.9449 

2 

Prepare  for  Pressure-Decay  Check  of  Turbo¬ 
pump  Gearbox 

II 

.9212 

.9444 

2 

Install  Gearbox  Pressurization  Kit 

I 

,9244 

.9469 

3 

Perform  Pressure-Decay  Check  of  Turbopump 
Gearbox 

I 

.9267 

.9492 

3 
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TABLE  1  (cont.) 


Human  Reliability 


Stage 

Kdl-  ■ 
vidual 

Redun- 
.  dant 

Criti¬ 

cality 

Prepare  for  Pressure-Decay  Check  of  Turbo¬ 
pump  Gearbox 

I 

.9327 

.9535 

2 

Prepare  for  Electrical  Check 

II 

.9337 

.9523 

3 

Perform  Pressure-Decay  Check  of  Turbopump 
Gearbox 

II 

.9350 

.9553 

3 

Perform  Visual  Inspection  of  Areas  Worked 
on  in  Test  Above,  to  Ensure  Hardware 
Integrity  and  That  all  Tools  Have  Been 
Removed  from  Area 

I 

.9W 

.9603 

3 

Ser/ice  Oil  Sump  of  Turbopump  Gearbox 

II 

.95^4 

.9686 

3 

Prepare  for  Turbopump  Torque  Check 

II 

.9575 

.9727 

3 

Prepare  for  Installation  of  Gearbox 
Pressurization  Kit 

I 

.9583 

.9728 

3 

Perform  Visual  Inspection  of  Areas  Worked 
on  in  Test  Above,  to  Ensure  Hardware 
Integrity  and  That  all  Tools  Have  Been 
Removed  from  Area 

II 

.9590 

.9794 

3 

Perform  Turbopump  Torque  Check  of  Sub- 
Subassembly 

I 

.9601 

.9751 

3 

Prepare  for  Installation  of  Gearbox 
Pressurization  Kit 

II 

.9609 

.97^7 

3 

Prepare  for  Subassembly  Turbopump  Torque 
Check 

I 

.983^ 

.9917 

2 
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II,  Summary  (cont.) 

C.  SUGGESTED  APPLICATIONS 

If  the  results  of  the  current  validation  study  at.  Vandenberg  Air  Force 
Base  are  favorable,  design  engineers  for  future  systems  may.  use  the  reliability- 
figures  derived  from  the  study  to  Improve  the  selection  of  components.  !Ehe  total 
reliability  of  a  system  is  dependent  on  the  interaction  of  its  components  with  per¬ 
sonnel,  Designers  may  prefer  to  select  a  component  of  somewhat  lower  hardware 
reliability  which  has  higher  human  reliability  in  operation  and  maintenance,  so  that 
the  man/machine  interaction  will  result  in  improved  system  reliability. 

Because  this  study  indicates  that  the  number  of  task  elements  critically 
deteimines  the  reliability  with  which  a  maintenance  task  will  be  performed,  the  num¬ 
ber  of  steps  required  to  operate  and  maintain  equipment  should  be  held  to  a  minimum. 

The  development  of  a  personnel  redundancy  formula  provides  a  convenient 
way  of  estimating  the  improvement  in  task  reliability  when  more  than  on  mechanic  is 
assigned  to  a  single  maintenance  task.  Opt.lmum  reliability  can  be  approached  if 
procedural  writers  give  careful  consideration  when  specifying  the  degree  and -manner 
of  interaction  of  mechanics  performing  a  tasK. 

To  determine  optimum  preventive  maintenance  schedules,  the  results  of 
the  present  study  could  be  used  with  the  information  derived  from  the  effects  of 
silo  storage  on  the  reliability  of  hardware.  Such  schedules,  by  incorporation 
quantitative  information  on  human  reliability  in  mathematical  models  of  system 
effectiveness,  would  give  realistic  consideration  to  the  personnel  subsystem. 

Thus  far  the  evidence  indicates  the  possibility  of  serious  problems  for 
planners  of  future  systems  of  extremely  high  reliability  requirements.  Tradition¬ 
ally,  the  reliability  of  systems  is  estimated  without  taking  the  reliability  of 
the  personnel  subsystem  into  account.  As  Williams  (Ref  6)  points  out,  the  product 
reliability  is  very  different  when  the  reliability  of  human  functions  is  used  in 
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computing  system  reliability.  The  present  study  offers  an  approach  to  acquiring  the 
necessary  quantification  of  human  behavior  that  is  essential  (if  the  human  element 
is  to  be  accounted  for)  in  estimating  system  reliability. 

One  final  caution  is  in  order.  Majesty  (Ref  9)  points  out  that  vhile  a 
human  reliability  metric  is  highly  desirable  and  vill  immeasurably  aid  engineers  in 
their  approach  to  designing  increasingly  reliable  systems,  human  feeling,  motiva¬ 
tions,  changes,  aberrant  reactions  under  stres,  etc.  must  also  be  considered.  Man 
shoald  not  be  viewed  simply  as  a  component  of  fixed  or  predetermined  reliability. 


Eeport  LEP,  317/5^-63-218 
III.  DEVELOPMEMT  OF  TBE  METHOD 

A.  PREVIOUS  EFFORTS 

This  section  contains  a  fairly  extensive  discussion  of  previous  efforts  in 
determining  human  reliability.  It  is  intended  for  readers  professionally  concerned 
with  this  problem.  Readers,  who  are  professionally  interested  in  other  fields  and  in 
a  detailed  description  of  the  method  finally  selected,  may  wish  to  turn  directly  to 
Section  III,  B. 


In  recent  years,  there  have  been  numerous  attempts  to  predict  and  measure 
human  reliability  (Ref  2,3,*4-,5>  and  6).  The  topic  received  considerable  attention 
in  the  program  on  personnel  subsystem  reliability  held  at  the  National  Aerospace 
Systems  Reliability  Symposium  in  Salt  Lake  City,  Utah,  l6  through  18  April  I962 
(Ref  7>8,9,  and  10). 

Recently,  Task  Group  2  of  the  Electronics  Industry  Association's  Military 
Subcommittee  on  Human  Factors  in  Electronics  was  established  to  achieve  two  goals ; 

(1)  to  design  a  system  to  collect  and  process  human-error  rate  data,  and  (2)  to  pre-  , 
pare  the  necessary  mathematical  models  and  procedures  for  predicting  the  degradation 
that  the  human  element  introduces  to  system  or  subsystem  effectiveness . 

Five  general  approaches  to  determining  human  reliability  cein  be  identified; 

» 

1.  Analysis  of  Field  Experience 

2.  Extrapolation  from  the  Experimental  Literature 

3 .  Conducting  Special  Studies  in  a  Simulated  Environment 

4.  Conducting  Special  Studies  in  the  Operational  Environment 

5.  Judging  and  Rating  Reliability 

These  five  general  approaches  to  determine  human  reliability  are 
discussed  as  follows: 
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III,  A,  Previous  Efforts  (cont.) 


1,  Analysis  of  Field  Experience 


Eie  first  and  most  common  approach  is  the  analysis  of  field  experJ.ence . 
During  normal  maintenance  operations,  empirical  data,  derived  from  accumulating  records, 
may  he  used  to  establish  error  rates .  Although  this  approach  appears  attractive ,  cer¬ 
tain  disadvantages  become  apparent  on  investigation.  Most  reporting  systems  are 
designed  to  report  hardware  malfunctions  but  give  inadequate  attention  to  desdriptioh 
of  human  errors .  In  addition,  such  reports  depend  heavily  on  the  accuracy  and  thorough¬ 
ness  of  the  reporting  personnel.  Often,  the  person  making  an  error  fails  to  notice  it 
or  is  unwilling  to  report  it.  Consequences  of  error  may  be  discovered  later,  but  the  ' 
analysis  of  cause  depends  on  inferences  about  probable  actions  leading  to  the  event. 

Such  limitations  are  so  serious  that  unless  special  precautions  are  taken,  field 
reports  are  misleading  in  establishing  degree  of  humein  error. 

For  example,  Shapero,  et  al.  (Ref  11),^  studied  3829  malfunction 
reports  from  seven  different  missile  test  programs  and  419  "unscheduled  hold"  reports 
from  two  missile-systems  test  efforts: 


a.  Of  the  3829  malfunction  reports,  39?^  vere  classified  as  human- 


initiated  . 


b .  Of  the  419  unscheduled  holds ,  205^  were  classified  as  human- 
initiated. 

c.  Most  important,  the  analysis  of  two  systems  in  which  the 
Ballistic  Missile  Division  Failure  and  Code  List  was  used  revealed  322  human-initiated 
malfunctions  in  one  system  ana  193  in  the  other.  Prior  to  the  analysis,  the  reporting 
personnel  (who  filled  out  the  original  failure  data  report)  classified  only  three 
human-error  incidents  in  one  system  and  none  in  the  other. 
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III,  A,  Previous  Efforts  (cont.) 


Field  reports  are  seriously  limited  because  they  cover  only 


unde,  .irablc  events .  Data  on  successes  or  total  number  of  attempts  are  difficult  to 


obtain.  To  establish  error  rate  or  hxanan  reliability,  little  is  gained  by  knowing 
that  a  percentage  of  malfunctions  or  failures  is  due  to  human  error  or  that  so  many 
human  errors  occurred  this  month  compared  to  the  number  reported  last  month,  or  a 
year  earlier.  To  determine  the  reliability  with  which  certain  tasks  are  performed 
by  mechanics,  it  is  necessary  to  know  the  number  of  times  a  task  was  attempted  and 
the  number  of  attempts  resulting  in  success.  A  final  limitation  is  that  such  data 
is  available  only  for  existing  systems .  To  predict  human  reliability  for  a  system 
in  the  early  stages  of  research  and  development,  performance  data  will  be  limited 
to  existing  systems  only.  It  may  be  possible  to  extrapolate  from  data  on  an  earlier, 
similar  system,  but  additional  problems  will  be  encountered. 


In  many  ways,  the  best  illustration  of  the  analysis  of  field 
experience  is  the  recent  study  by  Rook  (Ref  h)  for  the  Sandia  Corporation.  In  the 
field  of  nuclear  ordnance,  emphasis  is  on  hi^  reliability  and  a  "no-field- test" 
philosophy.  Critical  importance  is  placed  on  the  reduction  of  humeui  error  in  pro¬ 
duction.  Rook  was  able  to  collect  error  rate  data  for  certain  classes  of  human 
error  in  production,  such  as  "Two  wires  which  can  be  transposed  are  transposed"  and 
"Soldering  operation  results  in  insufficient  solder."  IheBe  rates  were  obtained  from 
an  analysis  of  over  23,000  production  defects  detected  in  assembly  operations  of 
electronic  equipment .  Rook  suggests  an  extremely  promising  classification  scheme 
that  will  aid  in  generating  design  and  procedure  change  proposals .  A  quantitative 
model  was  also  developed  for  evaluating  the  contribution  of  human  error  to  the 
degradation  of  product  quality. 


2.  Extrapolation  from  Experimental  Literature 


Recently  an  ingenious  approach  to  predicting  the  reliability  of 
operator  performance  was  developed  by  Payne  and  Altman  (Ref  1  and  3)  •  They  reasoned 
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that  during  15  to  20  years  of  experimentation,  most  significant  factors  affecting  per- 
f omance  were  identified  and  studied .  Ihey  surveyed  several  thousand  research  reports 
eind  finally  selected  l64  reports  meeting  their  requirements. 


To  make  the  findings  of  experimental  studies  useful  in  predicting 
the  operability  of  new  equipment,  it  was  necessary  to  develop  a  common  conceptual 
framework.'  The  framework  involved  23  components  such  as  circular  scales,  li^ts, 
cable  connections,  object  positioning,  etc.  For  each  component,  the  associated 
criteria  affecting  the  operator's  performance  were  identified.  For  example,  the 
criteria  affecting  performance  on  the  component  "lights"  were  size,  brightness,  type/ 
function,  number,  and  presentation. 


Data  were  abstracted  from  the  literature  in  this  framework.  Rational 
and  empirical  approaches  were  used  to  reduce  and  Integrate  the  data  and  render  the 
laboratory  data  more  compatible  with  field  conditions .  The  final  result  is  a  Data 
Store  that  provides  reliability  estimates  for  the  criteria  of  each  performance.  These 
data  can  be  combined  by  using  the  product  rule  to  obtain  reliability  estimates  at  the 
behavior  step,  mission  phase,  6ind  total  mission  levels. 

3.  Conducting  Special  Studies  in  a  Simulated  Environment 


Another  approach  for  determining  human  reliability  is  to  conduct 
special  studies  in  a  simulated  environment.  Applied  experimental  research  would 
make  the  subject's  task  as  similar  as  possible  to  the  anticipated  field  situation. 
Performance  data  from  the  experimental  sitmtion  are  used  to  predict  performance  in 
the  field  situation.  One  problem  is  the  difficulty  of  simulating  certain  situations 
at  reasonable  cost.  Also,  in  the  absence  of  field  data,  it  is  difficult  to  assess  the 
adequacy  of  the  simulation.  Knowing  that  the  situation  is  a  laboratory  rather  than  a 
field  situation  will  affect  the  subject's  pei’forraance  'thus  adding  to  the  difficulties. 
However,  'the  most  serious  limitation  of  this  approach  for  purposes  of  estimating  human 
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III,  A,  Previous  Efforts  (cont.) 


reliability  is  the  very  large  number  of  observations  needed  to  establish  reasonably 
accurate  reliability  estimates.  Payne  and  Altman  (Ref  l)^ estimate  that  mean* behavior 
step  unreliability  in  typical  field  situations  is  .0026.  Rook  (Ref  4)  obtained  error- 
rate  figures  varying  from  .07  to  .00003.  To  establish  such  rates,  hundreds  and  pre¬ 
ferably  thousands  of  observations  would  be  necessary. 


Conducting  Special  Studies  in  an  Operational  Environment 


Limitations  of  data  generated  from  routine  field  experience,  problems 
associated  with  assessing  the  relevsince  of  laboratory  data,  and  adequacy  of  simulation 
in  the  laboratory  approach  have  led  to  special  studies  in  the  operational  environment. 

A  recent  study  of  this  nature  is  repoj-ted  by  Meister  (Ref  12) .  IDie  study  involved 
analysis  of  702  Job  operations  during  Atlas  testing  at  Vanderiberg  Air  Force  Base.  The 
study  was  conducted  at  the  Operational  System  Test  Facility  (OSTF),  which  is  a  facility 
for  testing  the  adequacy  of  the  total  missile  system,  including  personnel  fuinctiohs. 
This  facility  is  a  complete  operational  missile  site  in  which  missiles  are  received, 
checked  out,  and  fired. 


The  702  Job  operations  were  observed  over  a  period  of  nine  months  by 
personnel  trained  to  observe  these  operations .  Observations  made  on  a  one-observer  to 
one-operator  basis  overcame  many  d3.sadvantages  inherent  in  the  operator-reported  errors  . 
0  f  field  experience  analysis .  Unfortunately,  for  the  purposes  of  the  present  study, 
Meister  does  not  report  the  rate  at  which  errors  occur  but  rather  the  percent  of  errors 
falling  in  certain  error  categories .  For  example ,  his  findings  suggest  that  the  largest 
proportion  of  the  observed  errors  are  "system"  rather  than  "operator"  errors .  System 
errors  include  such  things  as  nonavailability  of  personnel,  improper  personnel  utiliza¬ 
tion,  nonavailability  of  equipment,  nonavailability  or  inadequate  technical  data,  and 
inadequate  system  organization. 
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To  illustrate  how  the  probatility  analysis  techniques  utilized  for 
equipment  can  also  be  applied  to  the  analysis  of  human  factors  in  a  system,  Williams 
(Ref  6)  used  judges'  ratings  of  the  probability  of  success  for  various  operator 
actions  such  as  "Reading  frequency  from  a  moving  scale-type  dial,"  and  "Recording 
frequency  as  received  over  telephone."  Eatings  on  a  varie'cy  of  tasks  can  be  obtained 
quickly  and  inexpensively.  The  ratings  should  have  some  va].idity  if  the  raters  are 
thoroughly  familiar  with  kinds  of  tasks  being  rated  and  with  the  type  of  personnel 
performing  them.  Raters  were  asked  to  predict  absolute  reliability  directly  (.97# 

•99}  etc.).  Williams'  approach  required  Judges  to  estimate  figures  with  which  they 
had  little  experience  (human  reliability  in  performing  tasks) .  The  present  study 
utilized  a  combination  of  a  modified  rating  approach  and  empirically  based  reliabilities . 


B.  METHOD  OF  OBTAINING  RATINGS 


This  section  describes  the  method  utilized  to  obtain  likelihood-of-error 
ratings  in  the  performance  of  malntensuice  of  the  Titan  II  propulsion  System. 


Maintenance  Tasks  that  must  be  Performed 


To  determine  the  performance  reliability  for  each  silo  maintenance 
task,  the  activity  tasks  must  first  be  identified.  In  the  present  study.  Type  II  and 
Type  III  basic  data,  prepared  in  accordance  with  AFBM  Exhbiit  60-26A,  was  used  for 
this  purpose.  Two  broad  areas  of  maintenance  activities  were  considered.  First, 
scheduled  or  preventive  maintenance,  and  second,  unscheduled  or  corrective  mairitenance . 
To  date,  the  following  scheduled  maintenance  phases  have  been,  investigated: 


a .  Engine  servicing 

b.  Engine  visual  inspection 

c.  Engine  checkout. 
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III,  B,  Method  of  Obtaining  Ratings  (cont.) 

These  activities  were  further  factored  into  specific  tasks  such  as: 

a.  Service  oil  sump  of  turbopump  gearbox. 

b.  Perform  visual  inspection  of  areas  worked  to  ensure  hardware 
integrity,  and  that  all  tools  have  been  removed  from  the  area. 

c.  Prepare  for  electrical  check. 

d.  Perform  electrical  check. 

(The  full  set  of  tasks  is  presented  in  Table  l) 

2.  Identify  the  Task  Elements  Necessary  to  Accomplish  the  Total  Tasks 

Each  task  requires  specific  acts  that  are  smaller  units  of  human 
behavior  than  are  the  tasks  themselves.  Because  they  occur  repeatedly  in  a  variety 
of  tasks,  these  smaller  units,  or  task  elements,  lend  themselves  readily  to  an  analyaifl 
of  performance  reliability.  For  example,  the  activity  "Engine  Checkout"  includes  the 
task  "Prepare  for  Leak  Check  of  Turbopump  Fuel  Pump  Seal,"  which  contains  13  elements 
(Table  2) . 


These  elements,  although  related  directly  to  the  task  of  "Prepare  for 
Leak  Check  of  Turbopump  Fuel  Pump  Seal,"  are  repeated  in  other  tasks  such  as  "Prepare 
for  Leak  Check  of  Turbopump  Fuel  Gearbox  Seal."  Furthermore,  a  closer  analysis  of 
each  element  reveals  a  discrete  action  that  is  common  not  only  to  the  propulsion  system 
but  to  a  variety  of  mechanical  systems.  Therefore,  establishing  performance  reliability 
at  this  level  broadens  the  applicability  of  the  results .  An  ajialysis  was  made  of  the 
activity  tasks,  and  60  discrete  task  elements  (Position  Hand  Valves,  Install  "0"  Ring, 
Remove  Lockwire,  etc.)  were  selected  for  reliability  determination  (Table  9)- 
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TASm  2 

TASK  ELEMENTS  IN  TASK,  "PREPARE  FOR  LEAK  CHECK  OF  TURBOPUMP  FUEL  PUMP  SEAL" 


_ TASK  ELEMENT _ 

1.  Verify  that  all  valves,  regulators,  and  switches  are  in  proper  position,  and  that 
the  functional  test  set  is  otherwise  prepared  for  use. 

2.  Connect  a  valve  and  hose  assembly  to  the  "Fuel  System  Leak  Test"  port  on  the 
functional  test  set. 

3*  Remove  the  protective  closure  from  the  quick-disconnect  coupling  on  the  turbopump 
fuel  discharge  line. 

4.  Connect  the  valve  and  hose  assembly  to  the  quick  disconnect  coupling. 

5 .  Disconnect  the  fuel  pump  seal  cavity  drain  tube  assembly  "B"  nut  from  the  pump 
seal  cavity  drain  fitting. 

6.  Remove  the  cavity  drain  tube  assembly  from  the  engine. 

7 •  Install  the  reducing  adapter  on  the  cavity  drain  fitting . 

8.  Connect  a  flexible  hose  assembly  to  the  reducing  adapter. 

9 •  Remove  the  pressure  caps  from  the  flowmeter  inlet  and  outlet  ports . 

10.  Connect  the  flexible  hose  from  the  cavity  drain  to  the  flowmeter  inlet  port. 

11.  Disconnect  the  pressure  sequence  valve  overboard  drain  tube  assembly  "B"  nut 
from  the  overboard  drain  check  valve. 

12.  Remove  the  overboard  drain  tube  assembly  from  the  engine. 

13.  Install  a  pressure  cap  on  the  overboard  drain  check  valve. 
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3.  Determine  the  Performance  Reliability  for  Each  Task  Element 

Once  the  task  elements  have  been  identified,  the  reliability  of  the 
performance  of  each  by  a  missile  engine  mechanic  must  be  rated.  Ihe  present  approach 
differs  from  that  of  Rook  (Ref  h),  who  determined  the  rate  at  which  specific  types  of 
;er3rors  occur.  For  example,  in  "Soldering  a  wire,"  Rook  predicts  the  rates  at  which 
excess  solder,  insufficient  solder,  or  a  hole  in  the  solder  will  occur.  In  the  present 
study,  the  aim  is  to  predict  the  reliability  with  which  the  total  task  of  soldering  a 
wire  will  be  accomplished.  The  effort  to  predict  reliability  rather  than  associated 
error  rates  was  based  on  the  following  considerations:  all  possible  errors  are  not 
known,  whereas  required  task  elements  are  clearly  specified  in  Type  III  basic  data, 
technical  orders,  checklists,  and  other  procedural  documents.  Because  there  are  fewer 
task  elements  than  possible  errors,  the  judgisag  task  of  the  raters  is  simplified. 
Communicating  reliability  estimates  of  the  personnel  subsystem  in  familiar  terms  to 
engineers  working  on  hardware  subsystems  increases  the  liklihood  of  their  using  human 
reliability  information  in  problems  involving  man-machine  interactions.  Hie  approach 
has  the  disadvantage  of  leaving  the  nature  of  the  errors  unspecified. 

4.  The  Preliminary  Rating  Study 

To  test  the  feasibility  of  a  rating  approach  in  determining  performance 
reliability,  task  elements  were  sorted  into  a  forced  normal  distribution  (Figure  l) . 
Judges,  selected  for  their  knowledgeability  and  experience,  were  requested  to  sort  60 
task  elements  printed  on  l-l/4-in.  by  5-in.  cards  into  a  set  of  tied  ranks  in  accordance 
with  prescribed  directions  (Appendix).  Five  personnel-subsystem  engineers,  three  Air 
Force  missile-engine  mechanics,  and  one  Aerojet  engine  mechanic  were  appointed  Judges. 
Criteria  for  their  selection  included  knowledge  of  Titan  II  maintenance  tasks  smd  Air 
Force  engine-mechanic  performeince .  To  prevent  the  Judges  from  putting  many  cards  in 
only  one  or  a  few  categories  and  to  ensure  their  use  of  the  full  continuum,  a  forced 
normal  distribution,  such  as  used  by  Stephenson  (Ref  I3),  was  selected  for  the  prelimin¬ 
ary  study.  On  the  other  hand,  the  forced-choice  approach  has  the  disadvantage  of 
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III,  B,  Method  of  Obtaining  Ratings  (cont.) 

violating  the  assumption  of  independence  of  ratings.  If  a  rater  places  a  task  in 
Category  I  (least  error),  he  cannot  place  pother  task  in  that  category  even  thou^ 
he  believes  it  belongs  there. 

a.  Measurement  Reliability  of  Ratings 

Reliability,  as  the  term  is  used  by  psychologists,  refers  to 
the  accuracy  and  consistency  of  measurement.  It  should  not  be  confused  with  the 
engineering  use  for  referring  to  the  probability  that  a  component  or  system  will  per¬ 
form  successfully.  To  avoid  possible  confusion,  the  former  will  be  referred  to 
throu^out  this  report  as  "measurement  reliability,'.'  Hie  probability  that  the  personnel 
subsystem  will  perform  successfully  is  referred  to  as  "human  reliability"  or  "reli- 
ability  of  task  performance."  Ihe  probability  that  a  hardware  component  or  system 
will  perform  successfully  is  referred  to  as  "hardware  reliability." 

Using  the  intraclass  correlation  formula  recommended  for  such 
problems  by  Bbel  (Ref  1*4-),  the  measurement  reliability  of  tlie  nine  sets  of  ratings -was 
estimated.  Ihe  results  of  this  analysis  are  presented  in  Table  3*  Because  the  Intended 
use  of  the  obtained  ratings  will  be  based  on  average  ratings  of  task  elements  from  all 
nine  raters,  the  measurement  reliability  of  those  averages  are  the  most  Interesting. 

In  this  case,  the  measurement  reliability  was  .86,  indicating  a  high  degree  of  agree¬ 
ment  among  raters  concerning  the  likelihood  of  error  for  our  60  task  elements.  Of  - 
course,  this  did  not  establish  the  validity  of  such  ratings,  only  the  inter- rater 
measurement  reliability.  No  completely  valid  measures  of  the  human  reliability  of 
Titan  II  missile-engine  mechanic  task  performance  were  available.  However,  some 
empirically-based  measures  of  task-element  reliability  were  available  throu^  the 
Index  of  Electronic  Equiisnent  Operability  Data  Store  (Ref  3) •  From  the  Bata  Store 
(Table  4),  it  was  possible  to  obtain  human  reliability  estimates  for  29  of  the  60  tauk 
elements . 


Insert  illustrates  typical  task-element  sorting  card. 

Dotted  area  reprr-ents  forced  normal  sort  presented  to  raters 
during  preliminary  study. 


Sorting  Arrangement 


Figure  1 
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OlABLE  3 

MEASUREMENT  RELIABILITY  OP  RATINGS,  PRELIMINARy  STUDY 


Sum  of  squared  ratings 

= 

18,450 

Product  of  sum  and  mean 

2970 

= 

16,335 

Sum  of  squares 

Por  raters 

^  -16,335 

= 

0.0 

Por  task  elements 

^  -16,335 

= 

1,092 

Per  total 

18,450  -16,335 

= 

2,115 

Por  error 

2,115  -  1,092 

= 

1,023 

Mean  square 

Per  task  elements 

1092/59 

= 

18.5 

Per  error 

1023/472 

2.2 

Reliability  of  ratings 

18.5  -  2.2 

18.5  +  (9-l)2.5 

= 

.42 

18  S  -  2.2 

Reliability  of  average  ratings  — iS'~^' 

= 

.88 
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TABLE  4 

TASK  ECiMENT  RELIABILITIES  iROM  DATA  STORE 


Task  Element  Data  Store  Reliability 

Read  Electrical  or  ELov  Meter  .9860 
Read  Time  (Brush  Recorder)  *9873 
Read  Pressure  Gauge  *9897 
Tighten  Nuts,  Bolts,  and  Plugs  *9960 
Position  *'Zero-In"  Knob  .9962 
Connect  Electrical  Cable  (threaded)  *9968 
Install  Lock  Wire  .9968 
Install  0-Ring  . 9971 
Position  Multiple  Position  Electrical  Switch  *9972 
Install  Marman  Clamp  *9978 
Install  Gasket  .9978 
Position  Two  Position  Electrical  Switch  .9979 
Install  Reducing  Adapter  .9980 
Install  Nuts,  Plugs  and  Bolts  ,9980 
Install  Drain  Tube  .9980 
Install  Union  .9980 
Remove  Pressure  Cap  .9981 
Disconnect  Flexible  Hose  *9981 
Remove  Nuts,  Plugs  and  Bolts  *9981 
Install  Torque  Wrench  Adapter  *9981 
Remove  Reducing  Adapter  .9981 
Remove  Union  .  9981 
Remove  Drain  Tube  *9981 
Connect  ELexible  Hose  .9984 
Remove  Protective  Closure  (friction  fit)  .9990 
Install  Protective  Cover  (friction  fit)  >9990 

Remove  Torque  Wrench  Adapter  *9991 
Install  Funnel  or  Hose  In  Can  .9991 
Remove  Funnel  From  Oil  Can  .9991 
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ni,  B,  Method  of  Cfb talcing  Ratings  (cont.) 
b .  Results 

Figure  2  presents  the  scatter  plot  relating  ratings  and  Data 
Store  figures  for  the  29  tasks.  The  curvilinear  relationship  is  clear.  We  had 
expected  a  linear  relationship  and  wondered  if  the  obtained  relationship  might  have 
been  due  to  axi  artifact.  Some  raters  indicated  they  wanted  to  place  more  of  the  cards 
toward  the  least-error  end  of  the  continuum.  The  distribution  of  the  Data  Stofe 
figures  was  noticeably  skewed  in  that  direction.  Forcing  raters  to  normalize  what 
would  other  wise  have  a  skewed  distribution  of  ratings  would  result  in  the  sort  of 
curvilinear  relationship  obtained  in  Figure  2. 

Because  of  this  and  the  lack  of  independence  of  forced  ratings, 
it  was  decided  to  abandon  the  use  of  forced  ratings  eind  to  use  a  modified  Thurstone 
Method  of  Equal  Appearing  Intervals  procedure  in  the  main  study  (Ref  15), 

5.  Main  Rating  Study 

Based  on  the  experiences  gained  in  the  preliminary  study,  the 
decision  was  made  to  eliminate  the  use  of  the  forced  choice  but  to  retain  the  remain¬ 
der  of  the  rating  technique.  Judges  again  were  selected  on  the  basis  of  their  avail¬ 
ability  and  knowledge  of  Air  Force  techniciem  and  Titan  II  maintenance  tasks.  Identical 
task  elements  and  the  cards  used  in  the  first  study  were  employed.  A  modification  was 
made  in  the  instructions  as  follows: 

"You  have  been  selected  for  this  task  of  Titsm  II 
because  of  your  familiarity  with  the' .taaintenance  of 
Titan  II  engines,  plus  your  knowledge  of  the  capability 
of  Air  Force  personnel  to  perform  maintenance  activities. 

Please  use  your  knowledge  and  experience  in  estimating 
how  reliable  average  Air  Force  missile  engine  mechanics 
would  perform  a  set  of  selected  Jobs .  In  some  we  could 
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Error  Rate 


Scatter  Plot  of  Ratings  and  Data-Store  Error  Rates,  Preliminary  Study 


Figure  2 
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III,  B,  Method  of  Obtaining  Eatings  (cont. ) 

expect  almost  perfect  performance.  In  others  they  •will 
make  errors.  For  example,  in  one  study  more  errors  oc¬ 
curred  when  soldering  than  when  removing  wire  insulation. 

In  another,  more  errors  occurred  in  disconnecting  locking- 
type  cables  than  nonlocking -type  cables.  There  are  60 
cards  in  front  of  you,  each  with  a  different  -task  printed 
on  it.  These  are  the  types  of  ■tasks  that  a  missile  engine 
mechanic  will  perform  in  the  silo  during  main-tenance . 

On  the  "table  before  you  are  10  columns.  At  your  ex¬ 
treme  left  (Column  l)  place  the  items  you  believe  will  be 
performed  with  the  least  error  and  on  the  extreme  right 
(Column  10 )  place  the  items  that  will  be  performed  with 
the  most  error.  You  may  place  as  many  cards  as  you  wish 
in  any  of  the  columns.  As  you  move  from  left  to  right, 
each  column  represents  an  equal  increase  in  the  degree  of 
error.  If  at  any  time  you  wish  to  change  the  location  of 
any  cards,  feel  free  to  do  so." 

The  sorting  technique  was  administered  mainly  by  Aerojet  personnel 
spbsys'tem  and  reliability  human -factors  personnel.  In  some  cases,  the  ■task  was 
delega^ted  to  represen'tatives  in  "the  test  areas  at  Sacramento  or  at  Vandenberg  Air 
Force  Base. 


Eighteen  Aerojet  personnel  subsystem  and  reliability  engineers, 
six  Aerojet  technicians,  and  nine  Air  Force  missile -engine  mechanics  were  used 
as  judges  in  the  main  rating  study.  The  engineers  were  thoroughly  familiar  with 
maintenance  of  the  Titan  II  Propulsion  System.  They  were  knowledgeable  of  the 
engine  and,  from  ex^tensive  observation,  were  familiar  with  the  performance  of 
typical  Air  Force  mechanics.  The  Aerojet  technicians  were  familiar  with  the  -tasks 
involved,  having  themselves  performed  the  tasks  as  part  of  their  work  in  the 
Aerojet  -test  area  or,  in  one  case,  as  a  technician  in  the  Personnel  Subsystem  Test 
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III,  B,  Method  of  Obtaining  Ratings  (cont. ) 

and  Evaluation  Program.  The  Air  Force  mechanics  from  Vandenberg  Air  Force  Base 
were  familiar  with  the  tasks  and  were  participating  in  Category  II  testing  of 
Titan  II  during  the  time  they  served  as  raters  in  the  present  study. 

The  columns  into  which  the  judges  sorted  the  task  elements  were 
numbered  left  to  right  from  1  to  10  (Figure  l).  An  item  placed  in  a  column  received 
the  value  of  that  column.  When  sorting  was  completed,  the  cards  were  turned  over 
and  the  number  of  the  card  and  column  placement  was  recorded.  Thus,  each  card  re¬ 
ceived  33  independent  ratings  by  33  qualified  judges. 

a.  Measurement  Reliability  of  Ratings 

Because  the  raters  had  somewhat  different  backgrounds,  the 
reliability  of  ratings  was  investigated  by  sub-groups  of  raters  as  well  as  for  the 
entire  group  of  33  raters.  The  intra-class  correlation  for  estimating  the  reliabil¬ 
ity  of  ratings  (Ref  l4^  was  used  as  in  the  preliminary  study,  and  the  results  are 
presented  in  Tables  5>  6,  and  8. 

It  can  be  seen  that  the  most  reliable  ratings  were  obtained 
from  the  Aerojet  engineers  (.295  at  the  individual  rating  level  as  compared  with 
.263  for  the  Aerojet  technicians  and  .I30  for  the  Vandenberg  AFB  missile -engine 
mechanics).  Because  the  number  of  raters  differs  from  group  to  group,  the  indi¬ 
vidual  rating  level  is  the  most  appropriate  level  for  determining  the  most  reliable 
source  of  ratings.  The  measurement  reliability  of  average  ratings  based  on  I8 
raters  would  be  higher  than  the  reliability  of  average  ratings  based  on  6  raters 
even  though  the  ratings  were  fully  equivalent  in  other  respects. 

Inspection  of  the  distribution  of  ratings  reveals  the  reason 
for  the  relatively  low  measurement  reliability  of  the  technicians  and  Air  Force 
mechanics.  Whereas  l4  of  the  I8  Aerojet  personnel  subsystem  and  reliability  engi¬ 
neers  used  the  full  range  of  available  ratings  (from  1  to  lO),  only  one  of  the 
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!EABLE  5 

MEASUREMENT  RELIABILITY  OF  RATINGS— 33  RATEE?S 


Sum  of  Squared  Ratings 

= 

36,211 

Product  of  Sum  and  Mean 

(7003) (7003) 

24,769 

1980 

Sum  of  Squares 

For  Raters 

-  ^^,769 

= 

3,095 

'For  Tasks 

-  Sh,769 

= 

2,099 

For  Total 

36,211  -  2^,769 

= 

11,442 

For  Error 

11,442  -  3^095  -  2,099 

= 

6,248 

Mean  Square 

For  Tasks 

2,099  T  59 

= 

35.5763 

For  Error 

6248  f  1888 

3.3093 

Reliability  of  Ratings 

35.5763  -  3.3093 

.2281 

35.5763  +  32  (3.093) 

Reliability  of  Average  Ratings 

35.2542  ”  3.3194 

35.25^^2 

= 

.9153 
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TABLE  6 

MEASUREMENT  RELIABILITY  OP  RATINGS— l8  AGO  ENGINEERS 


Sum  of  Sqmred  Ratings 

= 

28,680 

Product  of  Sum  and  Mean 

(4900^^4900) 

= 

22,231 

Sum  of  Squares 

For  Raters 

1,352,316  -  22,231 

60 

s 

307 

For  Tasks 

437,120  -  22,231 
'  W 

= 

2,053 

For  Error 

6449  -  2053  -  307 

s: 

4,089 

Mean  Square 

For  Tasks 

2053  :  59 

s 

34.7966 

For  Error 

4089  7  1003 

s 

4.0767 

Reliability  of  Ratings 

34.7966  -  4.0767 

s; 

.2950 

34.7966  +  17  (4.6767) 

Reliability  of  Average  Ratings 

34.7966  -  4.0767 

= 

.8828 

34.7966 
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TABLE  7 

MEASUREMENT  RELIABILITy  OF  RATINGS— 6  AGO  TECHNICIANS 


Sum  of  Squared  Ratings 

= 

4,205 

Product  of  Sum  and  Mean 

(1003) (1003) 

360 

= 

2,794 

Sum  of  Squares 

For  Raters 

186,081  -  2,794 

66 

= 

307 

For  Tasks 

19^317  -  2,794 
— 

= 

426 

For  Total 

4,205  -  2,794 

= 

l,4ll 

For  Error 

l,4ll  -  307  -  426 

= 

678 

Mean  Square 

For  Tasks 

426  i  59 

= 

7.2203 

For  Error 

678  :  295 

s 

2.2983 

Reliability  of  Ratings 

7.2203  -  2.2983 

7.2203  +  (6  -  1;  2.2963 

= 

.2630 

Reliability  of  Average  Ratings 

7.2203  -  2.2983 

= 

.6816 

7.2203 
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TABIiE  8 

measurement  reliability  of  ratings— 9  VANDENBERG  APB  MISSILE -ENGINE  MECHANICS 


Sum  of  Squared  Ratings 

Si 

3,326 

Product  of  Sum  and  Mean 

(1100) (1100) 

540 

2,24i 

Sum  of  Squares 

For  Raters 

151.456  -  2,241 

60 

283 

For  Tasks 

21.808  -  2,241 

9 

182 

For  Total 

3,326  -  2,24l 

1,085 

For  Error 

1,085  -  283  -  182 

620 

Mean  Square 

For  Tasks 

182  59 

3.0847 

For  Error 

620  472 

1.3135 

Reliability  of  Ratings 

3.0847  -  1.3135 

3.084T  +  (9  -  1)  1.3135 

.1303 

Reliability  of  Average  Ratings 

3.084T  -  1.3135 

3.0847 

.5T4i 
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III,  B,  Method  of  Obtaining  Ratings  (cont. ) 

technicians  and  one  of  the  Air  Force  missile -engine  mechanics  did  so.  Three  of 
the  Air  Force  missile -engine  mechanics  and  one  of  the  Aerojet  technicians  assigned 
ho  ratings  higher  than  3.  All  engineers  used  ratings  from  1  to  9*  Those  using  a 
hi^ly  restricted  range  did  not  understand  the  instructions  or  they  perceived 
errors  as  much  less  likely  than  did  the  other  raters.  One  possibility  considered 
was  to  eliminate  some  of  the  raters.  Thurstone  (Ref  15),  using  an  eleven -point 
coniiinuum  and  100  to  125  items  to  be  rated, eliminated  raters  who  placed  30  or  more 
statements  in  a  single  category.  However,  the  validity  of  this  approach  has  been 
questioned  by  Hovland  and  Sherif  (Ref  l6),  who  found  in  one  study,  that  if  raters 
were  eliminated  according  to  this  criterion,  more  than  two-thirds  of  their  judges 
would  have  been  eliminated  and  genuine  differences  between  raters  obscured. 

In  a  sense,  the  Aerojet  technicians  and  the  Air  Force  mechan¬ 
ics  were  rating  themselves,  whereas  the  Aerojet  engineers  were  rating  others. 
Generally,  the  error  of  leniency,  which  is  a  problem  in  performance  ratings,  is 
particularly  pronounced  in  self-ratings.  Perhaps  the  present  finding  is  simply 
another  example  of  the  tendency  of  humans  to  be  unwilling  or  unable  to  report 
their  own  errors.  Perhaps  job  incumbents  are  not  good  raters  for  this  purpose. 

A  final  possibility  is  that  the  difference  in  ratings  is 
related  to  the  circumstances  under  which  the  xatings  were  requested.  The  Aerojet 
engineers  were  asked  to  perform  the  ratings  by  a  colleague.  Because  the  requester 
and  the  raters  were  working  in  the  same  department,  the  engineers  had  a  fairly  ade¬ 
quate  understanding  of  the  purpose  of  the  ratings.  In  the  other  two  cases,  the 
ratings  were  requested  by  an  "outsider,"  whose  purpose  was  less  likely  to  be  fully 
understood.  In  the  absence  of  a  fully  defensible  procedure  for  rejecting  "careless 
judges,  and  because  ratings  from  the  engineers  were  the  most  reliable  (.88)  and 
were  obtained  frcan  fully  qualified  judges,  who  were  not  ego-involved  in  the  ratings 
it  was  decided  to  base  the  remaining  analyses  on  those  ratings. 
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III,  B,  Method  of  Obtaining  Ratings  (cont. ) 


6.  Results 


The  means  and  standard  deviations  of  the  ratings  assigned  to  each 
task  element  are  presented  in  Table  9*  The  task  elements  are  arranged  in  order  of 
rank,  from  most  to  least  error.  Generally,  reading,  inspecting,  and  installing 
tasks  were  judged  more  likely  to  produce  error  than  removal  tasks.  Raters  showed 
the  greatest  agreement  (smallest  standard  deviation)  in  rating  these  tasks:  (l) 
Install  Funnel  or  Hose  in  Can,  (2)  Loosen  Nuts,  Bolts,  and  Plugs  (3)  Remove  Funnel 
from  Oil  Cah;  (4)  Remove  Union,  (5)  Remove  Drain  Tube,.,  and  (6)  Position  Two-Position 
Electrical  Switch. 


Raters  showed  the  least  agreement  (largest  standard  deviation)  in 
rating  these  tasks:  (l)  Read  Electrical  or  Flow  Meter,  (2)  Inspect  for  Bellows 
Distortion,  (3)  Lubricate  Bolt  or  Plug,  (h)  Inspect  for  QC  Seals,  (5)  Close  Hand 
Valves,  and  (6)  Tighten  Nuts,  Bolts,  and  Plugs. 

To  determine  the  relationship  of  the  present  results  to  the  pre¬ 
liminary  study,  or  the  results  that  would  have  been  obtained  using  Aerojet  tech¬ 
nicians  or  Air  Force  missile -engine  mechanics  as  raters,  Pearson  product -moment 
correlations  were  computed  between  the  mean  ratings  received  for  the  task  elements 
under  the  four  conditions.  These  results  are  presented  in  Table  10.  There  is 
substantial  agreement  among  the  four  groups  of  raters.  Similar  though  not  identical 
results  would  have  been  obtained  if  another  source  of  ratings  had  been  used  instead 
of  the  one  finally  selected. 

Figure  3  shows  the  mean  frequency  with  which  the  task  elements 
were  assigned  to  categories  by  the  l8  engineers.  The  distribution  is  skewed  un¬ 
like  the  forced  normal  distribution  used  in  the  preliminary  study. 
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TABLE  9 


TASK-ELEMENT  EELIABILm  ESTIMATES 


Task  Element 

Read  technical  instructions 

Read  time  (brush  recorder) 

Read  electrical  or  flow  meter 

Inspect  for  loose  bolts  and 
clamps 

Position  multiple -position 
electrical  switch 

Mark  position  of  component 

Install  lockwire 

Inspect  for  bellows  distortion 

Install  Marraan  clamp 

Install  gasket 

Inspect  for  rust  and  corrosion 

Install  "0"-ring 

Record  reading 

Inspect  for  dents,  cracks, 
and  scratches 

Read  pressure  gauge 

Inspect  for  frayed  shielding 

Inspect  for  QC  seals 

Tighten  nuts,  bolts,  and  plugs 

Apply  gasket  cement 

Connect  electrical  cable 
(threaded) 

Inspect  for  air  bubbles  (leak  check) 
Install  reducing  adapter 
Install  initiator  simulator 
Connect  flexible  hose 


Rating 
Mean  S.D. 


Individual 

Reliability 


.9901 

.990k 

.9928 

.9938 

.99kO 

.99kl 

.99kk 

.99kk 

.99*^*^ 

.99k^ 

.99k6 

.99k8 

.99k9 

.9950 

.9952 

.9952 

.9953 

.9953 

.995*+ 

.9955 

.9957 

.9958 

.9958 

.9958 


Percent  of  Redundant.  . 
Redundancy  Reliability 


.9979 

..9933 

.9949 

.9944 

.9952 

.9964 

.9955 

.9955 

.9972 

.9961 

.9957 

.9964 

.9959 

.9960 

.9962 

.9962 

.9962 

.9962 

.9963 

.9968 

.9970 

.9975 

.9975 

.9975 
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TABLE  9  (cont.) 

TASK«ELEMEMT  RELIABILm  ESTIMATES 


Rating 


Individual  Percent  of  Redundant.. 


Task  Element 

Mean 

S.D. 

Reliability 

Redundancy 

Reliability 

Position  "zero-in"  knob 

k.8 

1.6 

.9959 

10 

.9963 

Lubricate  bolt  or  plug 

h.7 

2.7 

.9960 

10 

.9964  . 

Position  hand  valves 

k.6 

1,6 

.9962 

10 

.9966 

Inst: .11  nuts,  plugs,  and  bolts 

k,6 

1.7 

.9962 

30 

.9973 

Install  union 

1.8 

.9962 

4o 

.9977 

Lubricate  "0"-ring 

2.5 

.9962 

10 

.9966 

Rotate  gearbox  train 

4.4 

2.0 

.9963 

20 

.9970 

Fill  sump  with  oil 

4.3 

1.6 

.9964 

30 

.9975 

Disconnect  flexible  hose 

4.2 

2.0 

.9965 

30 

.9975 

Lubricate  torque -wrench 
adapter 

4.2 

2.2 

.9965 

10 

.9968 

Remove  initiator  simulator 

4.1 

1.9 

.9966 

50 

.9983 

Install  protective  cover 
(friction  fit) 

4.1 

2.2 

.9966 

20 

.9973 

Read  time  (watch) 

4.1 

2.1 

,9966 

30 

.9976 

Verify  switch  position 

4.1 

1.9 

.9966 

20 

.9973 

Inspect  for  lockwire 

4.1 

2.1 

.9966 

10 

.9969 

Close  hand  valves 

4.0 

2.6 

.9966 

30 

.9976 

Install  drain  tube 

4,0 

2.1 

.9966 

4o 

.9980 

Install  torque -wrench  adapter 

3.9 

1.7 

.9967 

10 

.9970 

Open  hand  valves 

3.8 

2.6 

.9968 

30 

.9978 

Position  two-position 
electrical  switch 

3.8 

1.5 

.9968 

30 

.9978 

Spray  leak  detector 

3.7 

2.0 

.9969 

20 

.9975 

Verify  component  removed 
or  installed 

3.5 

2.4 

.9971 

40 

.9983 

Remove  nuts,  plugs,  and  bolts 

3.5 

1.7 

.9971 

30 

.9980 

Install  pressure  cap 

3.4 

1.6 

.9971 

30 

.9980 
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TABLE  9  ^cont.) 


task-element  REUABILm  ESTIMATES 

— .  "  . .  I 


Rating  Individual  Percent  of  Redundant 


Task  Element 

Mean 

Remove  protective  closure 
(friction  fit) 

3.2 

Rfemove  torque -wrench  adapter 

3.0 

Remove  reducing  adapter 

3.0 

►Remove  Marman  clamp 

3.0 

Remove  pressure  cap 

2.8 

Loosen  nuts,  bolts,  and  plugs 

2.8 

Remove  union 

2.7 

Remove  lockwire 

2.7 

Remove  drain  tube 

‘2.6 

Verify  light  illuminated 
or  extinguished 

'  2.2 

Install  funnel  or  hose  in  can 

2.0 

Remove  funnel  from  oil  can 

1.9 

S.D. 

Reliability 

Redundeuicy 

Reliability 

1.6 

.9973 

•  30 

.9981 

1.6 

.9974 

30 

.9982 

1.7 

.9974 

30 

.9982 

1.7 

■  .9974 

60 

.9990 

1.8 

.9975 

30 

.9982 

1.3 

.9975 

30 

.9982 

1.4 

.9976 

30 

.9983 

1.5 

.9976 

20 

.9981 

1.4 

.9976 

40 

.9986 

1.6 

.9979 

30 

.9985 

0.8 

.9980 

20 

.9984 

1.4 

.9980 

20 

.9984 
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III,  Development  of  the  Method  (cont.) 

C.  EMPIRICALLY  BASED  PERFORMANCE -RELIABILITY  ESTIMATES 

Before  the  ratings  could  be  integrated  into  system  reliability  and 
effectiveness  models,  it  was  necessary  to  convert  them  into  reliability  or  error- 
rate  figures.  If  a  task  received  a  mean  rating  of  6.3,  it  was  necessary  to  know 
what  that  represented  in  terms  of  probability  of  success  in  10,000  attempts.  Also, 
the  validity  of  the  ratings  could  be  better  judged  if  some  empirically  based  fig¬ 
ures  were  available  for  comparison. 

1.  Approaches  Considered 

Each  of  the  following  four  empirically  based  approaches  described 
in  the  introduction  was  considered:  (a)  analysis  of  field  experience,  (b)  extrap¬ 
olation  from  the  experimental  literature,  (c)  special  studies  in  a  simulated  en¬ 
vironment,  and  (d)  special  studies  in  the  operational  environment.  Aerojet-General 
utilizes  a  trouble  report  form  called  the  Quality/Reliability  Report  (QRR).  These 
reports  are  screened  to  identify  failures  resulting  from  human  errors.  On  com¬ 
pletion  of  the  investigation  of  a  human-error  incident,  a  Trouble  Investigation 
Report  (TIR)  is  issued  to  cognizant  departments  documenting  the  corrective  preven¬ 
tive  action  initiated.  QRR's  and  TIR's  provided  a  basis  for  identifying  the  number 
of  human  errors  of  various  types.  However,  information  on  the  number  of  task 
attempts  was  not  available.  Because  reliability  is  the  ratio  of  the  number  of 
successes  to  the  number  of  attempts,  it  can  not  be  computed  without  information  on 
the  number  of  attempts, 

An  unsuccessful  effort  was  made  to  develop  a  method  for  estimating 
the  number  of  times  maintenance  tasks  were  performed.  Such  information  could  have 
been  derived  from  engine  logs  if  sufficient  time  and  personnel  had  been  available; 
however,  this  was  not  tlie  case.  Because  the  Titan  II  was  in  an  early  stage  of 
weapon-system  development,  only  limited  field  data  were  available,  and  these  data 
were  confined  to  research  and  developaent  operations,  which  may  have  differed  in 
important  ways  from  maintenance  perfonned  in  the  silo  under  environmental  field 
conditions.  This  approach  was  finally  abandoned. 
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TABLE  10 


USTERCORRELATIONS  AMONG  MEAN  RATINGS  OP  TASK  ELEMENTS  BY  DIFFERENT  GROUPS  OF  RATEES 


Number 

Type 

Preliminary  Study  Subjects 

18  AGO 
Engineers 

6  AGO 

Technicians 

9  mmaaile 
Engine  Mechanicc 

9 

.85 

.61 

.58 

18 

AGO  Engineers 

— 

.7^ 

.61 

6 

AGO  Technicians 

— 

.58 

9 

AP  Missile  Engine  Mechanics 

.61 

.58 

i 
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III,  C,  Empirically  Based  Performance -Reliability  Estimates  (cont. ) 

The  Index  of  Electronic  Equipment  Operability;  Data  Store  (Ref  3) 
provided  a  ready  means  for  obtaining  empirically  based  reliability  estimates  for 
29  of  the  60  task  elements.  As  explained  in  the  introduction,  the  Data  Store  fig¬ 
ures  are  based  on  an  extrapolation  from  the  experimental  literature.  Because  this 
•was  the  main  approach  utilized  in  the  present  study,  it  will  be  described  more 
fully  in  succeeding  sections. 

A  third  approach  considered  was  that  of  conducting  special  studies 
in  a  simulated  environment.  As  part  of  Aerojet’s  Category  I  Personnel  Subsys'tem 
Test  and  Evaluation  (PSTE)  Program  for  Titan  II,  first-  and  second-s-tage  engines 
have  been  ins-balled  in  engine  demonstrators  to  simulate  an  operational  missile  and 
launch  silo  at  Aerojet's  Sacramento  plant. 

The  test  results,  using  Aerojet  technicians  (with  backgrounds 
similar  to  Air  Force  missile -engine  mechanics)  as  -test  subjects  performing  scheduled 
maintenance  activities  in  the  simulated  silo,  are  available  in  a  series  of  technical 
operating  reports  published  in  accordance  with  AFBM  Exhibit  60-20A  and  AFEM  Exhibit 
58-1.  The  activities  were  carefully  observed  by  human-factors  psychologists  and 
propulsion-sys'tem  main-tenance  engineers,  thus  overcoming  the  limitations  involved 
in  self-reported  errors.  However,  each  observed  maintenance  -task  was  performed 
only  a  few  times  by  a  limited  group  of  subjects.  Hundreds  or  thousands  of  additional 
performance  trials  would  have  to  be  scheduled  to  provide  dependable  estima-tes  of  per¬ 
formance  reliability.  To  do  this  at  the  task  level  would  have  been  extremely  costly 
and  required  many  additional  -test  subjects. 

Another  possibility  would  be  to  simula-te  the  performance  situation 
at  the  -bask  element  rather  than  at  the  full  task  level.  Ins'bead  of  observing  sub¬ 
jects  perfoming  leak  checks  of  the  fuel  system,  observe  them  reading  time  on  a 
Brush  Recorder,  ins-balling  lock  wire,  etc.  The  time  and  personnel  requirements 
would  be  considerably  reduced  by  working  at  the  elemental  rather  than  the  full 
■bask  level.  Each  trial  would  -take  minu-bes  instead  of  hours  to  complete.  A  dis- 
advan-bage  of  this  approach  ’,;ould  be  the  artificial  nature  of  the  -bask  situation. 
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III,  C,  Empirically  Based  Performance -Reliability  Estimates  (cont. ) 

Subjects  may  not  be  motivated  to  perform  in  the  same  vay  in  laboratory  situations 
as  under  more  realistic  circumstances.  Also,  the  reliability  with  which  a  task 
element  is  repeatedly  performed  mi#t  differ  significantly  from  the  reliability 
with  which  the  same  element  is  performed  when  it  is  only  one  element  in  a  sequence 
of  4o  elements.  Logically,  one  might  expect  that  errors  of  omission  would  be  more 
likely  under  the  latter  condition.  Although  information  on  the  possibilities  and 
limitations  inherent  in  the  task-element-simulation  approach  is  needed,  it  wss 
decided  not  to  attempt  such  a  study  as  part  of  the  present  effort. 

!Ibe  fourth  approach  considered  was  based  on  special  studies  in 
the  operational  environment.  An  attempt  was  made  to  obtain  copies  of  Meister's 
reports  (l^ef  12)  describing  the  analysis  of  maintenance  operations  at  an  Operational 
System  Test  Facility  at  Vandenberg  Air  Force  Base.  The  plan  was  to  translate  Atlas 
maintenance  activities  into  the  60  task-element  framework  and  obtain  empirically 
based  reliability  figures  (based  on  the  nine -month  period  of  observations  at 
Vandenberg  Air  Force  Base  during  which  observations  were  conducted  on  a  one  ob¬ 
server  and  one  operator  basis).  Unfortunately,  copies  of  the  detailed  reports 
will  not  bo  received  in  time  to  use  them  ir  the  present  study. 

As  an  alternative,  such  information  will  be  obtained  during 
Category  II  testing  of  the  Titan  II  at  Vandenberg  Air  Force  Base.  Because  the 
breakdown  of  task  elements  was  based  on  Type  III  data  for  Titan  II,  the  translation 
problem  was  eliminated.  Also,  because  Aerojet-General  is  developing  the  Titan  II 
propulsion  system,  ready  access  is  available  to  expert  knowledge  regarding  Titan 
maintenance.  This  is  not  possible  working  with  Atlas  data.  The  plan  for  obtaining 
performance  reliability  data  during  Category  II  testing  will  be  described  more  fully 
in  a  later  section. 

2,  The  Index  of  Electronic -Equipment  Operability 

The  major  source  of  empirically  basea  reliability  estimates  in 
the  present  study  is  the  Payne -Altman  Index  of  Electronic-Eq.uiianent  Operabilltjy. 
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III,  C,  Empirically  Based  Performance -Reliability  Estimates  (cont.) 


Interested  readers  should  consult  the  documents  describing  the  development  and 
use  of  the  Index  (Reference  1  and  3)*  To  illustrate  the  use  of  the  index  in 
estimating  rel.iability  (the  index  also  provides  estimates  of  time  required  to 
perform  tasks),  the  following  example  from  the  present  study  is  given: 


One  of  the  60  task  elements  is  "Read  electrical  or  flow  meter. 
To  estimate  reliability  for  this  element,  one  enters  the  Data  Store  under  the 
input  component,  Semicircular  Scale.  Reliability  figures  for  each  of  the 
following  parameters  are  determined: 


Paramet  er  Re lia  b i lit: 


Size 

c.  1-2  in.  .9993 

Scale  style 

b.  Quantitative  information  .9982 

(l)  Moving  pointer 

Parallax  (not  applicable) 


Scale  arc  length 

b.  50-100  .9950 

Scale  interval  spacing 

c.  l/lO  in,  less  than  l/2  in.  .9955 

Scale  brightness 
c.  Easily  perceptible  from 

normal  position  .9998 

Number  of  graduation  marks  per 

unit  of  required  resolution 

b.  Every  5th  unit  .9992 

Proportion  of  graduation  marks 
numbered 

b.  1:5  .9995 
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III,  C,  Empirically  Based  Performance -Reliability  Estimates  (cont.) 

Parameter  Reliability 

Scale  increase 

b.  Right  to  left  .9996 

Exposure  (viewing)  time' 

d.  Indefinite  .9997 

The  reliabilioy  of  reading  an  electrical  or  flow  meter  is  the  product  of  these 
separate  parameter  values,  in  this  instance  .9860. 

In  developing  the  index,  Payne  and  Altman  discovered  that  error 
estimates  obtained  from  the  experimental  literature  seemed  gross  overestimates 
of  operational  errors.  They  attributed  this  to  the  tendency  of  experimenters  to 
make  tasks  unusually  difficult,  or  to  count  near  errors  in  order  to  have  measurable 
error  without  running  a  large  number  of  trials.  Whatever  the  cause  of  the  difference, 
to  derive  operational  meaning  from  experimental  data,  it  was  necessary  to  adjust 
all  the  experimental  results  to  make  them  more  compatible  with  field  operation. 

Payne  and  Altman  did  so  in  the  following  manner:  Based  on  this  experience  (with 
a  variety  of  equipment)  of  the  American  Institute  for  Research,  they  estimated 
that  the  "mean  mission  failure  rate"  (roughly  analogous  to  the  task  failure  rate 
in  this  study)  was  I3  per  cent.  That  is,  it  may  be  said  that  about  13  per  cent 
of  the  time,  operator  error  will  fail  or  seriously  degrade  mission  effectiveness. 
Because  no  field  studies  had  been  conducted,  which  provided  reliability  estimates 
at  what  is  called  the  task  element  level,  it  was  necessary  to  derive  a  task-element 
reliability  from  the  task  reliability  estimate.  This  was  done  by  first  determining 
the  mean  number  of  steps  in  a  task.  A  veuriety  of  operating  manuals  for  electronic 
equipment  were  examined.  The  number  of  steps  required  to  operate  each  item  of 
equlianent  was  determined,  and  the  mean  number  of  steps  computed.  The  mean  number 
of  steps  was  approximately  50.  When  the  mean  mission  (task)  unreliability  estimate 
of  .13  was  divided  by  the  mean  number  of  steps,  a  mean  mission  step  (task  element) 
unreliability  estimate  of  .0026  was  obtained. 
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III,  C,  Empirically  Based  Performance -Reliability  Estimates  (cont.) 

This  step  (task  element)  unreliability  was  then  compared  with  an 
estimate  of  mean  unreliability  per  experimental  trial.  This  was  determined  from 
the  (abstracts)  data  available  from  experihiental  literature.  From  the  data 
available,  it  was  found  that  mean  unreliability  per  trial  was  .31935*  Thus,  there 
were  two  estimates  of  mean-step  unreliability,  one  based  on  actual  field  operation 
and  one  based  on  laboratory  experimentation.  Assuming  that  experimental  trials 
are  roughly  eguivalent  to  individual  steps  of  operation,  the  ratio  of  these  means 
is  a  reasonable  conversion  factor  for  laboratory  results.  Accordingly,  Payne 
and  Altman  corrected  all  experimental  results  by  a  factor  of  or  .008l45. 


a.  Modified  Conversion  Factor 


For  the  purpose  of  this  study,  it  was  decided  that  it  may  be 
possible  to  derive  a  more  appropriate  conversion  factor  by  utilizing  the  results 
of  the  Personnel  Subsystem  Test  and  Evaluation  (PSTE)  studies  conducted  at  the 
simulated  silo.  All  studies  conducted  during  I962  were  examined.  During  this 
period,  974  tests  were  conducted. 

The  following  partial  list  of  tasks  will  indicate  the  level 
of  description  involved: 

(1)  Prepare  for  leak  check  of  oxidizor  system 

(2)  Perform  leak  check  of  oxidizer  system 

(3)  Prepare  for  leak  check  of  turbopump  oxidizer  pump  seal 

(4)  Perform  leak  check  of  turbopump  oxidizer  pump  seal 

(5)  Prepare  for  leak  check  of  turbopump  oxidizer  gearbox  seal. 

(6)  Perform  leak  check  of  turbopump  assembly  oxidizer  pump 
gearbox  seal 
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III,  C,  Empirically  Based  Performance -Reliability  Estimates  (cont.) 

Human  errors  that  concern  the  present  study  occured  on  l40 
(or  14.4^5^)  of  the  974  tests.  Thus,  the  task-level  reliability  was  .8560.  To 
determine  the  mean  task-element  level  reliability,  the  mean  number  of  task 
elements  per  task  was  determined.  This  was  42.  Becaus.e  it  was  planned  to  use 
the  product  rule  in  estimating  task-level  reliability  from  task-element  reliabil¬ 
ities,  the  procedure  was  reversed  in  the  present  instance.  The  task-element 
reliability  was  determined.  Wlien  raised  to  the  . 42nd  power  (representing  the'. 

42  task-element  steps),  it  would  give  a  task  reliability  of  ,8560,  This  was 
.9963*  The  estimate  of  task-element  reliability  of  .99^3  was  similar  to  the 
Payne-Altman  estimate  of  mean -step  reliability  under  field  conditions  of  .9974  (mean 
unreliability  =  .0026). 

Because  figures  obtained  using  the  Index  of  Electronic 
Equipment  Operability;  Data  Store  (Reference  4)  would  represent  slight  over 
estimates  of  the  reliability  to  be  expected  from  Titan  II  missile -engine  mechanics, 
each  task-element -level  estimate  was  reduced  by  the  following  conversion  factor; 


In  effect,  this  meant  reducing  each  index -derived  task-element  figure  by  .0011, 
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THE  RELATIONSHIP  BETWEEN  THE  RATINGS  AND 


ESTIMATES 


EMPIRICALLY  BASED  REUABILITy 


The  29  task  elements  for  which  both  ratings  and  modified  Index  of  Electronic 
E(itiipment  Operability  (Ref  3)  figures  were  available  were  plotted  on  a  scatter 
^agram  (Figure  k) .  The  Pearson  product -moment  correlation  between  these  two 
variables  is  .457>  which  is  significant  between  the  .0.5  and  .01  levels 
of  confidence.  Inspection  of  Figure  suggested  that  a  curvilineeur  relationship 
would  fit  the  data  better  than  a  linear  one.  The  data  were  replotted  on  semi- 
logarithmic  paper  with  the  results  shown  in  Figure  5* 

A  logarithmic  curve  was  fitted  to  these  data.  The  equation  for  the  loga¬ 
rithmic  curve  in  Figure  4  is  log  E  -  2.917^  -  .006122R,  vrtiere  E  =  error  rate  and 
K  =  pooled  rating  of  likelihood  of  error.  This  equation  was  used  to  derive  relia¬ 
bility  estimates  for  each  of  the  60  ratings.  These  estimates  are  presented  in 
Table  9. 

The  logarithmic  curve  was  selected  in  preference  to  a  linear  function  for 
two  reasons.  First,  the  fit  was  better.  Both  linear  and  logarithmic  regression 
equations  were  computed  relating  empirically  derived  error  rates  and  pooled  ratings 
The  sum  of  squared  deviations  from  the  linear  regression  line  was  .OOOl61<-19j  the 
sum  of  the  squared  deviations  from  the  logarithmic  regression  line  was  .00015242, 
a  reduction  of  7-2  percent.  Second,  the  logarithmic  relationship  has  a  long  his¬ 
tory  in  the  stj:tdy  of  psychophysics,  the  study  of  human  perception  of  physical 
magnitudes . 

The  Weber-Fechner  law  (Ref  17)  (S  ■  k  log  R)  states  that  sensation  (per¬ 
ception  of  brightness,  loudness,  weight,  etc.)  is  a  function  of  the  log  of  the 
magnitude  of  the  physical  stimulus.  While  not  a  universal  "law,"  this  relation¬ 
ship  holds  approximately,  except  for  extreme  values  of  the  stimulus.  The  present 
resxilts  are  consistent  with  the  Weber-Fechner  lav.  Sensation  (perception  of  the 
llkillhood  of  error  by  raters)  is  a  function  of  the  log  of  the  physical  proba¬ 
bility  of  error. 
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V.  TAKU^'G  PERSOMREL  BEDUKDANCY  INTO  ACCOUNT  IN  ESTB-ATING  TA^  REUABiUTY 


Previous  studies  of  human  reliability  have  been  limited  to  the  behavior 
of  individuals  operating  independently.  Because  the  Air  Force  directs  that  a 
technician  must  be  accon^ianied  by  at  least  one  other  individual  when  entering 
the  silo,  the  direct  application  of  data  acquired  by  such  investigations  is  some¬ 
what  inappropriate  for  estimating  the  reliabiiity  of  Titan  li  missile-engine 
mechanics.  Because  the  second  individual  is  likely  to  be  a  missile  engine  mechanic 
also,  his  presence  should  be  considered  in  the  manner  as  one  considers  red\a?dant 
equipment . 

The  principle  of  redundancy  is  commonly  applied  by  design  engineers. 

Dewing  (Ref  l8)  defines  redundancy  as  "the  existence  of  more  than  one  means  for 
accomplishing  a  given  task  where  all  means  must  fail  before  there  is  an  overall 
system  failure."  "Functional  redundancy,"  he  states,  "applies  to  systems  where 
two  or  more  means  are  working  at  the  same  time."  The  latter  definition  is  appro¬ 
priate  for  the  purpose  of  this  study.  If,  for  example,  one  operating  power  gen¬ 
erator  failed,  the  standby  generator  would  immediately  begin  operation,  and 
required  electrical  power  would  continue  to  be  transmitted.  If  the  reliability 
of  a  single  generator  is  estimated,  then  the  redundancy  reliability  of  the  two 
can  be  calculated  as  follows: 

E,  =  1  -  (1-Ei)” 

where  R^  is  the  redundancy  reliability 

Rj^  is  the  reliability  of  one  power  generator,  and 

n  is  the  total  number  of  generators. 


If  the  reliability  of  one  generator  is  .90  then  the  equation  would  be  as  follows: 

R  =  1  -  (l-.90)^ 

=  1  -  (.10)^ 

=  1  -  .01 

=  .9900 
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V,  Taking  Personnel  Redundancy  into  Account  in  Estimating  Task  Reliability  (cont.) 

In  estimating  human  reliabilities,  a  modification  of  the  hardware  redun¬ 
dancy  assumption  is  appropriate.  Unlike  the  generator,  a  second  individual  may 
not  always  be  "available"  to  "back  up"  the  first  individual.  To  assume,  however, 
that  he  would  never  be  available,  would  be  equally  inappropriate.  Therefore, 

When  two  men  are  working  together  to  perform  a  maintenance  task,  their  redundancy 
reliability  may  be  expressed  as  follows; 


1  -  (l.R^)“  (T^)  +  ^.(Tg) 


where  T^^  is  the  time  (in  that  the  second  man  is  in  a  position  to  react 
to  a  potential  error  of  the  first  man  (redundancy  time),  and  Tg  is  the 
remaining  time  (lOO^  " 

If  the  same  figiires  are  used  for  human  reliability  as  were  used  previously 
in  the  example  of  generator  redundancy,  changing  only  the  percentage  of  redundancy 
time,  the  results  should  be  .9OOO  but  .9900. 


R  =  39.6  +  54.0 

^r  100 

Ry  =  .9360 


For  the  purpose  of  the  above  exan5>le,  hO  and  60^  were  selected  arbitrarily. 
However,  when  applied  to  specific -problem  estimating,  it  is  essential  that  time 
percentages  be  estimated  by  knowledgeable  Judges. 
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y,  Taking  Personnel  Redundancy  into  Account  in  Estimating  Task  Reliability  (cont.) 

In  the  present  study,  ratings  of  percentage  of  redundancy  time  (by  10^ 
intervals)  vere  made  independently  by  two  engineering  psychologists.  On  83.3 
percent  of  the  60  task  elements,  they  disagreed  by  no  more  than  one  interval, 
on  eight  they  differed  by  two  intervals,  and  on  two,  by  three  intervals.  The 
Pearson  product -moment  correlation  between  the  ratings  was  .5^.  In  applying 
the  redundancy  formula  in  the  present  study,  the  average  of  the  independent 
ratings  used  was  rounded  to  the  nearest  ten-percent  interval.  Table  9  presents 
the  percentage  of  redundancy  figures  used. 

VI.  CONVERTING  TASK-ELEMENT  RELIABILITIES  INTO  TASK  REHABILITIES 


The  estimated  reliability  for  each  Titan  II  scheduled  maintensmce  task 
is  presented  in  Table  1.  These  figures  were  derived  by  taking  the  reliability 
estimates  for  task  elements  from  Table  9  and  applying  them  to  Type  III  basic  data. 
The  reliability  for  the  task  is  the  product  of  the  reliability  of  the  task  ele¬ 
ments.  To  Illustrate  the  method  involved,  one  of  the  shorter  tasks  is  presented 
in  detail  in  Table  11.  The  circled  reliabilities  in  Table  11  are  reliabilities 
for  Type  III  basic  data  task  "Prepare  for  Turbopump  Check."  Because  two  mechanics 
will  normally  be  assigned  to  a  single  task,  the  most  appropriate  estimate  of  the 
reliability  of  this  task  is  .9727*  This  fig’n*e  was  computed  by  determining  the 
product  of  all  the  task-element  reliabilities  in  Table  11.  Each  of  the  "Redun¬ 
dant"  reliability  figures  was  derived  from  the  corresponding  "Individual"  figure, 
using  the  redundancy  formula  described  in  Section  V. 

There  are  assumptions,  which  may  not  be  fully  warranted,  regarding  the 
use  of  the  product  rule.  For  a  discussion  of  this,  see  Section  X,  C,  More 
Elaborate  Treatment  of  Dependency. 
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VII.  A  COMPARISON  OF  RELIABILITY  ESIIMATBS  WITH  AIR  FORCE  MAINMIAMCE  RECORDS 

A  highly  promisiiag  source  for  obtaining  empirical  data  .Tor  validation  of  re¬ 
liability  estimates  is  Air  Force  operationsd.  maintenance  data.  The  commander  of  an 
Airborne  Early  Warning  and  Control  Wing  (AEW&C)  of  the  Air  Defense  Command  cooper¬ 
ated  with  the  Aerojet  research  team  by  providing  information  concerning  maintenance- 
inspection  activities  in  support  of  RC-121  radar  aircraft.  A  Quality  Control  Officer 
of  the  above  orgeinization  provided  records  of  performance  of  thousands  of  inspections 
performed  by  aircraft  mechanics.  These  records  were  carefully  checked  by  quality 
control  personnel.  The  tasks  were  compared  for  their  similarity  to  the  list  of  60 
task  elements  described  in  Table  9.  From  the  list  of  tasks  submitted  by  the  Air 
Force  unit,  the  following  three  were  selected -as- being ',directly..related  to  the  pres¬ 
ent  study:  (l)  "Inspection  of  exhaust  clamps,”  (2)  "Inspection  of  ignition  system," 
and  (3)  "Inspect  for  dents,  cracks,  and  damage  of  the  power- recovery  turbine." 

Inspection  of  the  exhaust  clamps  entails  the  visual  inspection  and  physical 
msuaipulation  of  the  engine  exhaust  system  to  ensure  the  security  of  the  bolts  emd 
clamps  (adjustment  is  necessary  if  they  are  insecure  or  loose).  The  part  is  then 
reinspected  by  the  quality- control  inspector.  This  task  is  made  up  of  elements  very 
similar  to  "Inspect  for  loose  bolts  and  clamps."  Reliability  for  the  Air  Force  task 
was  .9998  compared  to  the  estimated  reliability  of  .99^4  of  the  present  study.  The 
Air  Force  task  similar  to  the  Aerojet  element,  "Inspect  for  lockwire,"  was  the 
"Inspect  ignition  system."  In  addition  to  inspecting  for  loose  wires,  ignition; 
leads,  etc.,  the  presence  of  lockwire  is  checked.  Once  again,  the  empirical  error 
rate  is  similar  to  the  estimated  rate  (l09/l4,202  or  .OO76)  compared  with  an  esti¬ 
mate  of  I-.9944  or  .0056  in  the  present  study.  Because  other  errors  entered  into 
the  computation,  the  reliability  for  "lockwire"  might  be  higher,  but  no  lower,  than 
,9924.  The  task  "Inspect  for  dents,  cracks,  and  damage  of  the  recovery  turbine"  is 
similar  to  our  "Inspect  for  dents,  cracks,  and  scratches."  The  Aerojet  reliability 
estimate  is  .9950>  while  the  Air  Force  maintenance-inspection  reliability,  based  on 
l4  errors  in  Jdk  attempts,  was  .9921. 
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VII,  A  Comparison  of  Reliability  Estimates  with  Air  Force  Maintenance  Records  (cont.) 


Such  data  seem  to  justify  confidence  in  the  present  approach  and  support  the 
idea  that  the  technique  of  deriving  estimates  of  reliability  based  on  ratings  of 
human  performance  is  both  useful  and  accurate.  The  evident  potential  of  such 
sources,  as  the  Air  Force,  to  supply  empirical  data  in  support  of  reliability  esti¬ 
mates  is  encouraging.  It  has  been  difficvilt  in  past  to  identify  sources  of  such 
data. 


Because  the  amount  of  data  available  was  limited  to  only  3  of  the  60  elements, 
providing  congruent  rather  than  predictive  veLLidity,  plans  were  made  for  validating 
the  reliability  estimates  using  Titan  II  test  data. 

VIII.  PLAN  FOR  VALIDATION  OF  RELIABILITY  ESTIMATES 


The  final  test  of  the  method  developed  herein  is  a  comparison  of  predictions 
with  error  data  collected  in  the  Titan  II  silo  during  the  performance  of  Air  Force 
maintenance.  There  is,  at  present,  a  program  at  Vandenberg  Air  Force  Base  that  will 
provide  the  needed  data.  In  the  program.  Category  II  Personnel  Subsystem  Test  and 
Evaluation,  the  performance  of  missile-engine  mechanics  is  observed  and  reported. 

For  a  period  of  about  one  year,  each  scheduled  maintenance  task  will  have  been  ob¬ 
served  approximately  four  to  twelve  times.  Individual  Summary  Forms  (iSF's)  are 
written  at  the  conclusion  of  each  perforrasuice  indicating  the  observer's  comments  on 
the  quality  of  performance  (Figure  6). 

There  are  two  particularly  appropriate  ingredients  in  the  program  that  lend 
it  to  a  validation  of  the  method.  First,  the  ISF  reports  cover  "no  deficiencies" 
as  well  as  "deficiencies".  This  permits  the  use  of  a  ratio  that  can  be  expressed 
as  a  reliability  estimate.  That  is,  if  eui  airman  opened  30  valves  without  error, 
his  reliability  in  the  performance  of  opening  valves  could  be  e^qiressed  as  30/32  or 
•9375*  Secondly,  the  ISF  data  collected  during  this  program  are  based  on  observa¬ 
tions  of  performance  rather  than  on  inspections  following  perf ormeuxce . 
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VIII,  Plan  for  Validation  of  Reliability  Estimates  (cont.) 

The  reliability  estimates  were  derived  from  observations  of  performance,  so 
it  is  appropriate  that  they  be  validated  by  data  derived  from  observation. 

Inspections  reveal  error  that  remains  in  a  system  after  the  task  has  been 
completed,  while  "over  the  shoulder"  observations  reveal  errors  that  occur  during 
performance.  These  errors  are  consequential,  though  corrected,  because  they  may 
result  in  an  .inci:ease  in  maintenance  t^me,  personal 'injury,'  and  use  of  spare  parts. 

The  Individual  Summary  Forms  provide  information  in  a  manner  that  requires 
minimal  conversion  for  the  purposes  of  data  reduction  (Figure  6). 

In  the  sample  provided,  the  task  was  "Perform  Stage  I  Thrust- Chamber  Pressure 
Hot-Gas  System  Leak  Check  and  Thrust- Chamber  Pressure-Switch  Functional  Check."  By 
reviewing  the  procedures  in  Technical  Order  (T.O.)  21-SM68B-CL-2-1,  the  number  of 
required  lubrications  can  be  counted  and  recorded.  This  number  becomes  the  "total 
number  of  attemps."  The  deficiency  indicates  two  errors:  the  omission  of  lubricant, 
and  the  application  of  inappropriate  lubricant.  Therefore,  the  "number  of  errors"  is 
two.  Each  time  the  task  is  accomplished  at  Vandenber  Air  Force  Base,  both  the  niunber 
of  attempts  and  the  number  of  errors  are  counted.  At  the  conclusion  of  the  Category 
II  program,  a  reliability  will  be  computed  of  each  element  for  comparison  with  the 
estimates . 

IX.  SUGGESTIONS  FOR  FURTHER  RESEARCH 

A.  DETERMINING  PREVENTIVE  MAINTENANCE  SCHEDULES 

Results  of  the  present  study  could  be  used  to  help  resolve  the  conflict 
between  those  who  advocate  frequent  and  extensive  checkout  eind  maintenance  of 
Titan  II  engines  at  operational  sites  and  those  who  advocate  a  "hands  off"  policy 
once  the  missile  is  in  the  silo.  The  answer  to  the  problem  depends  on  the  trade-off 
between  degree  of  reliability  loss  of  a  ready  system  through  time  euid  the  reliability 
gain  or  loss  resulting  from  more  frequenv  checkout  and  mainteneuice . 
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FIGURE  6 

INDIVIDUAL  SUMMARY  FORM 


Perform  Stage  I  Thrust- Chamber 
Pressure  (P^)  Hot-Gas  System  LeaJc 
Titl£  Check  and  TOPS  Functional  Check 
Date  Observed  1  May  &  2  May  19b3 
Phase/Section  2 

T.O.  No.  &  Rev.  21-ai68B-a-2-l  (3?) 

Function/Para/Pig .  No .  10  ( 3o) 

Observer (s)  John  Doe _ 

Test  Location  VAFB 


Deficiency 

Classification 

Criticsd 

Major 

Minor 

Retest  Required 
None 


Observer 


Data  Eval. 
Group 


AFSC:  Specified 
443XLE  (2) 


il43XLE  (2) 


DEFICIENCY  CATEGORY: 


Time:  Specified 


Fixing  High-Press.  Leaks 
in  V-99.1  Item  1101. 3/ 
TTU-188/e 


END  HEMS  OP  EQUIPMENT 


Performance 


S/n  9510034  s/A1  &  S/A2 


DEFICIENCY  DESCRIPTION: 


Leakage  was  detected  by  soap  check  on  l/h"  hard-line  ”B”  nut  from 
TCPS  to  the  injector  boss  union.  The  union  was  removed  from  the 
injector  boss.  Upon  reinstallation  the  technicians  did  not  use 
thread  lubrication  on  the  union  end  into  the  injector.  They  used 
incompatible  lube  on  the  "B"  nut  connection  at  the  union  ’’B”  nut 
end. 


OBSERVER  RECOMMENDATIONS: 


Provide  all  compatible  thread  lubricants  in  the  work  area  and  observe 
thread-lubrication  practives  recommended  by  the  engine  manufacturer. 
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IX,  A,  Determining  Preventive  Maintenance  Schedules (cont.) 

Because  only  limited  data  are  presently  available  concerning  effects  of 
storage  on  assembled  Titan  II  propulsion  subsystems,  consideration  should  be  given 
to  the  use  of  adaptive  checkout  and  replacement  policies  in  vhich  the  interval  be- 
t¥een  checkout  and  replacement  is  adjusted  to  new  information  as  it  accumulates 

(Ref  19) . 


Results  of  studies  of  the  effects  of  silo  storage  on  the  reliability  of 
the  Titan  II  engines  should  be  considered  in  conjunction  with  the  results  of  the 
present  study  to  establish  an  optimal  preventive  maintenance  schedule.  Mathematical 
models  similar  to  those  provided  by  Kamins  (Ref  20)  are  needed  to  establish  schedules 
that  will  maximize  readiness  and  minimize  cost. 

B.  MORE  EXTENSIVE  COVERAGE  OF  MAINTENANCE  TASKS 

The  present  report  covers  scheduled  maintenance  and  checkout  tasks  only. 
The  appioach  could  be  extended  to  cover  unscheduled  maintenance,  trouble  shooting, 
and  depot  and  factory- level  tasks.  One  advantage  of  the  task  element  approach  is 
that  only  a  few  additioned  task  elements  are  needed  to  cover  many  additional  tasks 
of  a  similar  nature.  For  example,  a  preliminary  inspection  suggests  that  only  I5 
new  tasks  elements  are  needed  to  cover  most  unscheduled  mainteneuice  tasks.  These 
are  as  follows: 

1.  Verify  retaining  screw  in  place 

2.  Inspect  for  clesinliness 

3.  Clean  surface 

4.  Align  pin  holes 

5 .  Install  pin 

6.  Install  retaining  ring 

7.  Remove  retaining  ring 

8.  Remove  pin 

9.  Cut  electrical  leads 

10.  Attach  pull  wire  to  leads 
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IX,  B,  More  Extensive  Coverage  of  Maintenance  G?asks  (cont.) 

Pull  leads  through  conduit 
Install  crimp  connector 
Adjust  nut 

Install  identifying  tag 
Remove  identifying  tag 

C.  MORE  ELABORATE  TREATMENT  OF  DEPENDENCY 

The  use  of  the  product  rule  assumes  that  the  performance  of  each  element 
is  independent  of  the  performance  of  other  elements  in  the  task.  The  effect  of  de¬ 
pendence  has  been  to  some  extent  accounted  for  in  an  "average"  manner  by  applying  the 
rule  to  element  performance.  There  is,  however,  a  need  to  know  more  than  "average" 
interaction.  In  some  cases,  the  interaction  of  preceding  and  succeeding  elements 
results  in  significant  changes  in  either  direction  from  the  "average"  error  rate. 

For  example,  in  the  illustration  in  Table  10,  an  access  cover  cannot  be  removed  until 
the  lockwire,  bolts,  and  washers  have  been  removed.  If  the  mechanic  forgets  to  re¬ 
move  the  lockwire,  he  is  likely  to  recall  it  during  succeeding  steps.  Interaction 
may  eilso  effect  an  increase  in  error.  A  mechanic  may  be  required  to  remove  a  pro¬ 
tective  cover  from  a  port  before  connecting  a  cable  to  that  port.  Should  the  wrong 
port  have  been  uncovered  it  is  possible  that  the  cable  will  be  connected  in  error. 

A  comprehensive  study  of  interaction  is  needed  to  facilitate  accurate  determinations 
of  its  effect  on  the  reliability  of  perf ormeince . 

D.  MORE  EXTENSIVE  TREATMENT  OF  CONSEQUENCES  OF  ERRORS 

The  present  approach  attempts  to  account  for  errors  that  occur  during 
maintenance.  Those  errors  discovered  and  corrected  by  the  missile-engine  mechanic 
or  by  an  independent  inspection  scheduled  after  the  original  task  performance  are 
discounted.  Also,  the  treatment  of  the  criticality  of  errors  in  the  present  report 
is  recognizably  at  a  gross  level.  The  present  approach  was  taken  deliberately  in 
the  belief  that  all  errors  are  undesirable.  Even  though  corrected  at  a  later  time, 
an  error  may  have  caused  personed.  injury,  increase  in  maintenance  time,  or  consump¬ 
tion  of  spare  parts.  Though  all  errors  are  consequential,  it  would  be  unrealistic 


11. 

12. 

13. 

14. 
1'5. 
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IX,  D,  More  Extensive  Treatment  of  Consequences  of  Errors  (cont.) 

not  to  recognize  that  some  errors  are  more  serious  than  others.  Future  refinements 
of  the  method  should  take  these  factors  into  account. 

One  such  factor  that  may  lead  to  a  more  comprehensive  understanding  of 
the  natiore  of  error  is  the  distinction  between  acts  of  omission  and  commission.  At 
the  inception  of  the  study,  the  raters  were  instructed  to  make  judgements  of  likeli¬ 
hood  of  error.  Discussions  with  these  judges,  after  the  completion  of  their  task, 
revealed  their  desire  for  more  specific  descriptions  of  error.  The  statements  most 
commonly  heard  were  regarding  "things  which  are  done  wrong"  as  opposed  tc  "things 
which  one  forgets  to  do." 

The  purpose  of  this  study  was  not  to  dissect  error  but  to  develop  a 
means  of  predicting  the  rate  of  its  occurrence.  However,  there  is  value  in  further 
reseeirch  of  errors  of  omission  and  commission  because  the  consequences  of  these  may 
vary.  If,  for  example,  a  mechanic  forgets  to  open  a  veilve,  there  may  be  no  more 
consequence  than  a  loss  of  time.  He  must  correct  his  error  to  continue  the  task 
requiring  the  valve  to  be  opened.  If,  on  the  other  hand,  the  mechanic  had  opened 
the  wrong  valve,  then  he  may  have  permitted  fluids  to  enter  an  area,  thus  incurring 
degradation.  It  can  also  be  argued  that  certain  cases  of  omission  may  be  of  greater 
consequence  when  they  occur  because  they  are  not  brought  to  the  attention  of  the 
mechanic  by  a  succeeding  task.  Once  again,  if  a  veO-ve  was  erroneously  left  closed, 
it  is  possible  that  the  initiation  of  sui  action,  which  required  the  valve  be  opened, 
could  Induce  severe  damage  to  personnel  or  equipment  as  well  as  abort  the  task. 

Another  distinction  contributing  to  a  better  understanding  of  the  nature 
of  error  is  that  made  between  "Intentional"  and  "unintentionjd"  error.  Intentional 
error,  as  suggested  by  Rook  (Ref  4),  is  "one  of  conscious  awareness  by  the  operator 
....  the  operator  intends  to  perform  the  act  correctly,  but  erroneously  performs  it 
out  of  limits."  Unintentional  error  is  where  "there  is  no  element  of  intent  in  the 
perfomance  of  the  act  -  it  just  happens." 
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IX,  D,  More  Extensive  Treatment  of  Consequences  of  Errors  (cont.) 

If,  for  example,  a  mechanic  solders  a  wire  to  an  incorrect  connecting 
point,  he  has  committed  an  intentional  error.  However,  if  his  soldering  iron  inad¬ 
vertently  brushed  against  a  wire  removing  its  insulation,  he  would  have  committed 
an  unintentional  error. 

It  should  be  concluded,  therefore,  that  more  must  be  known  about  the 
nature  of  error  so  that  both  the  reduction  of  its  rate  and  the  prediction  of  its 
consequences  cein  be  attempted  with  success.. 

E.  DEVELOPMENT  OF  METHODS  FOR  RAPIDLY  ESTIMATING  TASK  RELIABILITY 

Occasions  arise  when  a  simple  approximation  of  the  reliability  for  a 
task  is  needed.  In  the  present  study,  close  approximation  to  the  present  results 
could  be  obtained  by  taking  the  mean  task-element  reliability  of  .9963  and  raising 
it  to  a  power  corresponding  to  the  number  of  task  elements  in  a  task.  Rough  relia¬ 
bility  estimates  for  tasks  (for  which  Type  III  Basic  Data  are  available)  could  be 
generated  very  quickly  by  estimating  the  mean- element  reliability  and  determining 
the  number  of  task  elements  in  each  task. 

Further  research  could  explore  the  adequacy  of  this  and  other  simple 
approximations . 
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GLOSSARY 


GAIEGORY  II  OaSTIIlG  (SYSTEM  DEVELOrMEHT  !EEST  ARP  EVALUATION)  —  Development  testing 
and  evaluation  of  integrated  subsystems  through  the  mating  process  that  progresses 
into  a  complete  system.  Conducted  to  determine  the  functional  capability  of  subsys¬ 
tems  and  redesign  requirements.  A  joint  contractor  Air  Force  effort  during  -vriiich 
the  Air  Force  effort  becomes  predominant. 

HARDWARE  RELIABILITY  — .  The  probability  that  a  hardvare  component  or  system  will 
perform  successfully. 

HUMAN  RELIABILITY  —  The  probability  that  the  personnel  subsystem  will  perform  suc¬ 
cessfully.  Individual  reliability  refers  to  human  reliability  when  the  task  is 
performed  by  a  single  person,  redundant  reliability  when  performed  by  two  persons. 

MEASUREMENT  RELIABILITY  --  The  consistency  and  accuracy  with  which  something  is 
measured. 

METHOD  OF  EQUAL  APPEARING  INTERVALS  —  A  scaling  method  adapted  by  L.  L.  Thurstone 
for  the  measurement  of  attitudes.  In  this  method,  judges  sort  a  large  and  represen¬ 
tative  pool  of  statements  about  an  object  into  groups  separated  by  equed  steps  of 
intervals.  The  average  of  their  judgements  defines  the  scale  value  of  a  statement, 

NORMAL  DISIRIBUTION  —  A  unimodal  symmetrical,  bell-shaped  frequency  distribution. 

PEARSON  PRODUCT  MOMEyP  CORREMTION  COEFFICIENT  —  A  statistic  for  describing  the 
degree  to  which  two  (or  more)  variables  are  so  related  that  a  change  in  one  is  ac¬ 
companied  by  a  corresponding  change  in  the  other.  Perfect  correspondence  is  ejtpres- 
sed  as  +1.00,  perfect  inverse  correspondence  as  -1.00,  and  complete  independence  of 
the  variables  is  expressed  as  0.00. 

PERSONNEL  SUBSYSTEM  --  That  major  functional  part  of  a  system  that,  through  effec¬ 
tive  implementation  of  the  various  elements,  provides  the  human  performance  neces¬ 
sary  to  operate,  maintain,  and  control  the  system  in  its  intended  operational 
environment. 

REDUNDANCY  --  A  design  factojj  for  the  purpose  of  increasing  the  probability  of  suc¬ 
cess,  that  Introduces  more  than  one  means  for  accomplishing  the  given  task. 

IffiGRESSlON  EQUATION  --  A  formula  for  computing  the  most  probable  value  of  one  vari- 
able  from  the  known  value  of  other  variables. 

SKEWED  DISTRIBUTION  --  A  nonsymmetrical  distribution,  where  the  items  with  the 
largest  frequencies  cluster  close  to  one  end  of  the  curve,  as  opposed  to  a  normal 
distribution, 

TASK  --  A  group  of  related  task  elements  performed  within  a  work  cycle. 
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Glossary  (cont.) 


TASK  RT.EMENT  --  A  single  action  (perception,  decision,  or  response)  that  a  person 
is  required  to  perform  in  the  completion  of  a  task. 

TYPE  II  BASIC  DATA  (Operation/Maintenance  Activities  Analysis)  --  Ihis  analysis 
translates  weapon- system  functions  in  terms  of  equipment  and  personnel  required  in 
the  performance  of  each  identified  weapon- system  activity.  The  data  are  given  in 
tabular  form  listing  derived  equipment  and  personnel  performance  chsiracteristics 
against  system  function.  Prepared  in  accordance  with  AFBM  Exhibit  60-26a. 

TYPE  III  BASIC  DATA  (Performance  Standards  (Proficiency)  Analysis)--A  detailed 
description  at  the  task-element  level  of  the  performeuice  of  tasks  identified  in  the 
Type  II  analysis.  Prepared  in  accordance  with  AFBM  Exhibit  60-26a. 

WEBER-FECHNER  LAW  --  A  psychophysical  law  showing  that  the  difference  in  a  r.timulus 
just  barely  noticeable  is  a  constant  proportional  part  of  the  original  stimulus. 
Thus,  human  perception  of  brightness,  loudness,  weight,  etc.  is  a  function  of  the 
log  of  the  magnitude  of  the  physical  stimulus. 
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APPENDIX 

INSTRUCTIONS  TO  RATERS  USED 
IN  PRELIMINARY  STUDY 


Report  LRP  317/tdR- 63-218,  Appendix 


"It  Is  generally  recognized  that  man  is  not  a  perfect  machine.  In  the  per¬ 
formance  of  some  tasks  he  makes  many  mistakes,  while  in  others  he  can  be  e:q)ected  to 
make  very  few.  You  are  being  asked  to  assist  us  in  determining  just  how  well  or  how 
poorly  man  performs  various  tasks.  Others  have  investigated  this  question,  suid  ex¬ 
amples  from  their  studies  may  give  you  some  idea  of  what  we  eure  seeking  to  find." 

■"For --example,  in  a  study  of  electronic  equipment,  it  was  shown  that  error  oc¬ 
curred  more  often  in  a  soldering  operation  than  in  removing  insulation  from  a  wire. 
Ih  smother  study  it  was  shown  that  mechanics  made  more  errors  in  disconnecting  lock¬ 
ing-type  cables  than  in  disconnecting  nonlocking  type  cables."" 


"On  the  table  before  you  are  ten  categories  of  task  errors.  Category  I  rep¬ 
resent  the  least  error,  and  Category  X  represents  the  most  error.  There  are  60  cards 
in  front  of  you,  each  with  a  different  task  printed  on  it.  These  are  the  types  of 
tasks  th|at  a  missile  engine  mechanic  will  perform  in  the  silo  during  scheduled  main¬ 
tenance.  Note  that  only  one  card  may  be  placed  in  Category  I,  three  in  Category  II, 
six  in  Category  III  and  so  forth.  Assume  that  missile -engine  mechanics  had  performed 
each  of  the  tasks  1,000  times.  Place  the  card  in  the  category  you  believe  represents 
the  relative  degree  of  error  that  probably  occurred.  If,  for  example,  you  think  that 
a  certain  task  could  have  had  so  little  error,  then  this  task  would  be  put  in  Cate¬ 
gory  I.  If,  on  the  other  heind,  you  believe  a  task  could  have  been  performed  with 
considerable  error,  for  example,  80  times  out  of  the  1,000  attempts,  and  that  no 
other  task  could  have  resulted  in  as  much  error,  then  this  task  would  be  placed  in 
Category  X.  If,  as  your  participation  progresses,  you  wish  to  change  any  of  your 
judgements,  feel  free  to  do  so." 
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